MeeGo wiki - User contributions [en]

Architecture/planning/evolution-data-server/eds-improvements

2011-10-06T06:53:24Z

Murrayc: /* More Efficient Access to Meta Data for Change Tracking */

This page describes improvements that are desirable or even essential for good performance of Evolution Data Server (EDS) as PIM storage in MeeGo.

All of these changes should be developed for the evolution-data-server gnome-2-32 branch, as used by MeeGo 1.2, then ported to master for review by upstream maintainers. Most changes will not be accepted in upstream's gnome-2-32 branch, because is is stable, but Meego will apply them as patches to its package.

See the list of [https://bugzilla.gnome.org/buglist.cgi?status_whiteboard_type=allwordssubstr;query_format=advanced;field0-0-0=product;status_whiteboard=meego;bug_status=UNCONFIRMED;bug_status=NEW;bug_status=ASSIGNED;bug_status=REOPENED;bug_status=NEEDINFO;type0-0-0=substring;value0-0-0=Evolution-Data-Server "meego" bugs in the upstream evolution-data-server bugzilla].

== Change Tracking ==

EDS is based on the model/view/controller principle: an application opens a database (calendar or addressbook) and requests a view with server-side filtering, then the application is sent:
# all data covered by the view
# the "all data sent" signal
# changes (add/update/remove) while the view exists

Applications modify data with asynchronous requests (controller) and keep showing the old data until the change notification comes in. The exact changes made to the data are determined by the server (for example when meeting invitations are sent).

QtContacts and KCalCore do not have such a strict model. With QtContacts the app simply loads and stores contacts and is responsible for maintaining its own view. Changes are sent for changes made after the database was opened. KCalCore is based on the concept of loading a calendar (partially) into the client, modifying it there, then writing it back. Changes are also sent while the storage is open.

TODO:
* https://bugs.meego.com/show_bug.cgi?id=18760 - "QtContacts-EDS + EDS: more flexible EBookView (UID only, only future changes)"

=== Suppress Unnecessary Data Transfer ===

EDS currently sends all data when a view is requested, which is unnecessary when the client is only interested in change notifications. An extension to the EDS query language could optionally suppresses that sending of existing data when the view is opened, to be implemented only for the local contacts and calendar backends. This would allow easier integration with the QContact API.

The Fremantle EDS fork also had a “freezable” (static) capability for libebook, which should probably be investigated. See, for instance: http://maemo.org/api_refs/5.0/5.0-final/libebook/EBookView.html#e-book-view-set-freezable

One possibility for adding the flag is to (ab)use the existing query language with a match for special fields:
* in the client: query = e_book_query_orv(e_book_query_field_exists("X-EDS-QUERY-FLAG-SKIP-DATA-DUMP"), <real query>, NULL);
* in the server: "if (query is OR-Query and first clause is exists("X-EDS-QUERY-FLAG-SKIP-DATA-DUMP")) { skip_data_dump = TRUE; query = second clause; }

[https://bugzilla.gnome.org/show_bug.cgi?id=652171 Upstream evolution-data-server bug]

=== Notify with UIDs Only ===

EBookView emits signals when contacts are changed, added, or removed. But those signals sends lists of entire EContacts, containing the entire contact data, though we often have all the data already and just want to know which contacts have changed.

[http://developer.gnome.org/libebook/stable/EBook.html#e-book-get-book-view e_book_get_book_view()] takes an [http://developer.gnome.org/libebook/stable/libebook-e-book-query.html#EBookQuery EBookQuery] which could be extended to take a new flag to make the EBookView send only the UID in the signals' EContacts. For instance, "X-EDS-QUERY-FLAG-NOTIFY-UID-ONLY". The UID is already stored separately for use as the primary key of the local-storage backend's BDB database, so no VCard parsing should be necessary to provide it.

This would allow easier integration with the QContact API, greatly improving performance.

Note that users of this feature should probably use just the base EVCard API on the provided EContact objects, to avoid extra processing and caching in the EContact code.

Note: We previously considered implementing e_book_get_book_view()'s requested fields parameter, but that changes the whole view, though we only really want to change what is sent for changed/new contacts.

Note: Chris discovered a dedicated e_book_get_contact_uids() API in the Fremantle EDS fork. The new API call was accepted upstream (in master) and [https://bugzilla.gnome.org/show_bug.cgi?id=651446#c11 backported as a patch] (used in the Meego package) for the gnome-2-32 branch, but this is not a full solution.

[https://bugzilla.gnome.org/show_bug.cgi?id=652172 Upstream evolution-data-server bug] (DONE)

== Optional Parsing of ebook Data in Client ==

The EDS client-side APIs currently parse and (temporarily) store the parsed details of VCards. However, this is often unnecessary because APIs such as e_book_get_book_view() cause it to return the whole VCard anyway. There will be some performance gain from delaying VCard parsing until it is really needed by calls to functions such as e_contact_new_from_vcard().

libebook provides different functionality:
# exchange data as vCard with EDS daemon
# parse items and wrap them in a C API

QtContacts replaces the second part. An obvious performance improvement is to only do the second part if the additional C APIs are really called. The following calls then transfer data without any additional parsing in libebook:
* e_book_get_contact()/asynchronous read/view + e_vcard_to_string()
* e_contact_new_from_vcard() + e_book_add_contact()/e_book_commit_contact()/asynchronous storing

Note that libecal (for iCalendar) can already do this, though libebook can't yet.

TODO:
# add delayed parsing to libebook, triggered by API call which needs parsed data

PROBLEM:
# If delayed parsing fails, how can error be reported?
# Should delayed parsing be done always (e_vcard_to_string())) or only if requested? How would that be configured by a libebook user?

An [http://build.meego.com/package/view_file?file=EBookView-Implement-delayed-vcard-parsing.patch&package=evolution-data-server&project=eds&srcmd5=977d2b885bed818cb616599749407e79 e_book_view_set_parse_vcards() patch], already used in the Meego package, partly implements this already. This patch was taken from an old fremantle EDS patch that was never actually used, and is generally considered to be a bad hack that should not be accepted upstream.

[https://bugzilla.gnome.org/show_bug.cgi?id=652173 Upstream evolution-data-server bug]

== Asynchronous API for storing items in libecal ==

libecal should allow the caller to request storing of data without blocking on processing of the data. It should notify the caller when the data has been stored, or when processing has failed.

These functions may require async versions:
* icalcomponent_new_from_string()
* e_cal_modify_object[_with_mod]
* e_cal_create_object()

Making libical asynchronous is almost impossible without going multithreaded. Let's focus on e_cal D-Bus communication first.

Rob started some work on this in the rbradford-wip-ecal-async-api branch. Perhaps you can reuse some of that work. It is not functional, but might be a source of inspiration or code fragments.

[https://bugzilla.gnome.org/show_bug.cgi?id=652174 Upstream evolution-data-server bug]

== More Efficient Access to Meta Data for Change Tracking ==

This was discussed a while ago: [[http://www.mail-archive.com/evolution-hackers@gnome.org/msg03029.html
[Evolution-hackers] concurrent modifications of items in GUI and EDS database]]

Data synchronization only needs to know which items exist (list of local IDs) and which revision of each item is currently stored. Revisions can be identified with a string which changes for each modification. Restoring an older revision should (but doesn't have to) restore the revision string.

For calendar, UID+RECURRENCE-ID and LAST-MODIFIED can be used. For contacts, UID + REV.

By comparing a stored list against the current one, changes can be detected without relying on journals in the daemon or comparison of time stamps. Journals are hard to maintain (how many of them, how long?). Comparison of time stamps has problems when changes are allowed while a sync runs and assumes that system time is linear (which is not always true).

As an additional use case, local ID + revision string could be used to prevent overwriting stale data in the server, as in HTTP PUT eTag checks.

TODO:

The Maemo Fremantle port of EDS, which is of course already open source, apparently has some solution for this, according to Mathias Hasselman, so that should be investigated first.

However, this should probably be made possible by:

# Extending the EDS query language with a flag that limits the delivered data to just:
## UID and REV for libebook (REV is in VCard 3.0)
## UID+RECURRENCE-ID+LAST-MODIFIED, for libecal
# Implementing these queries for the local contact and calendar backends. The query would be rejected, with an error, for other backends.

[https://bugzilla.gnome.org/show_bug.cgi?id=652179 libebook: Upstream evolution-data-server bug] (DONE) 
[https://bugzilla.gnome.org/show_bug.cgi?id=652180 libecal: Upstream evolution-data-server bug]

== Atomic Updates ==

Later, implement atomic changes as discussed in the mail thread: [http://www.mail-archive.com/evolution-hackers@gnome.org/msg03029.html]

== Quick check for "data changed" ==

The libebook and libical APIs should offer a way to discover if any data whatsoever has changed. This would, for instance, help SyncEvolution to avoid unnecessary detailed checking. This could just check the modification dates of local storage files, such as the BDB .db file, or .ics file, as long as we do not use those files to store permanent caches of parsed data. This would be much quicker than listing items for the whole database and comparing their revisions strings, as is currently necessary.

This would require small API additions, to check that the opaque revision strings are unchanged, such as

gchar* e_book_get_revision(EBook *book)
gboolean e_book_check_revision(EBook *book , const gchar* revision)

and

gchar* e_cal_get_revision(ECal *ecal)
gboolean e_cal_check_revision(ECal ecal, const gchar revision)

The semantic would be this:
* If the revision string is empty or different, then the content of the database '''might''' have changed. A more detailed check is necessary to determine if it has really changed. This will be the case for non-local storage.
* If it is non-empty and the same as before, the nothing has changed. No further check will be necessary.

[https://bugzilla.gnome.org/show_bug.cgi?id=652175 libebook: Upstream evolution-data-server bug] 
[https://bugzilla.gnome.org/show_bug.cgi?id=652177 libecal: Upstream evolution-data-server bug]

== Contacts: Store PHOTO Data as Plain Files ==

Photos should be transferred over D-Bus via a filepath to a local file rather than transferring the actual photo data over D-Bus. The VCard format already allows this option (See the PHOTO property). We must be careful to keep these local files in sync and to remove them when contacts are removed, and to restrict access to the files from other processes.

There should be utility code available to libebook users to convert between the different representations.

Goals:
# Apps should be able to create photo files without ever passing the data to libebook.
# External PHOTO file and corresponding contact must be kept in sync:
## If a contact is deleted, the photo also needs to be removed.
## If a photo file is created and then storing the contact fails, who removes the file?
# Access to photo data must be limited to processes which have the right permissions.

Proposal:
* For each ~/.local/share/evolution/foo calendar, store photos in <photo path>=~/.local/share/evolution/foo/photos, with same permissions.
* New photo files are created with a [http://en.wikipedia.org/wiki/Universally_unique_identifier UUID] as base name and suffix according to the file content.
* If a contact is deleted and has a PHOTO;VALUE=uri:file://<photo path>/*, delete that file.
* For apps which want to create the photo file directly:
** Apps get a file descriptor and the path from a new libebook API.
** They own the file until the EDS server acknowledges the successful storing of a contact with a matching PHOTO;VALUE=uri:file property. If that fails, the app must remove the file.
* Add new API calls to libebook which split out resp. include data inline.
** Must track ownership of file after splitting it out and before storing - might never get stored!

Open:
* Should there be automatic garbage collection of stale photos in case something went wrong and a photo wasn't properly deleted?
* Trust TYPE=JPEG/PNG/... or look into file content to determine type?

[https://bugzilla.gnome.org/show_bug.cgi?id=652178 Upstream evolution-data-server bug]

== Future: Content Protection ==

Only authorized applications (= processes) with the necessary permissions should be able to read/write data. Exact permissions and mechanisms for verifying them to be defined as part of the (revised?) MeeGo architecture.

This can be added as follows:
# lock down file permissions
# add permission checks to EDS D-Bus servers

Currently not planned. Depends on the future MeeGo security infrastructure.

Architecture/planning/evolution-data-server/eds-improvements

2011-10-06T06:50:50Z

Murrayc: /* Notify with UIDs Only */

This page describes improvements that are desirable or even essential for good performance of Evolution Data Server (EDS) as PIM storage in MeeGo.

All of these changes should be developed for the evolution-data-server gnome-2-32 branch, as used by MeeGo 1.2, then ported to master for review by upstream maintainers. Most changes will not be accepted in upstream's gnome-2-32 branch, because is is stable, but Meego will apply them as patches to its package.

See the list of [https://bugzilla.gnome.org/buglist.cgi?status_whiteboard_type=allwordssubstr;query_format=advanced;field0-0-0=product;status_whiteboard=meego;bug_status=UNCONFIRMED;bug_status=NEW;bug_status=ASSIGNED;bug_status=REOPENED;bug_status=NEEDINFO;type0-0-0=substring;value0-0-0=Evolution-Data-Server "meego" bugs in the upstream evolution-data-server bugzilla].

== Change Tracking ==

EDS is based on the model/view/controller principle: an application opens a database (calendar or addressbook) and requests a view with server-side filtering, then the application is sent:
# all data covered by the view
# the "all data sent" signal
# changes (add/update/remove) while the view exists

Applications modify data with asynchronous requests (controller) and keep showing the old data until the change notification comes in. The exact changes made to the data are determined by the server (for example when meeting invitations are sent).

QtContacts and KCalCore do not have such a strict model. With QtContacts the app simply loads and stores contacts and is responsible for maintaining its own view. Changes are sent for changes made after the database was opened. KCalCore is based on the concept of loading a calendar (partially) into the client, modifying it there, then writing it back. Changes are also sent while the storage is open.

TODO:
* https://bugs.meego.com/show_bug.cgi?id=18760 - "QtContacts-EDS + EDS: more flexible EBookView (UID only, only future changes)"

=== Suppress Unnecessary Data Transfer ===

EDS currently sends all data when a view is requested, which is unnecessary when the client is only interested in change notifications. An extension to the EDS query language could optionally suppresses that sending of existing data when the view is opened, to be implemented only for the local contacts and calendar backends. This would allow easier integration with the QContact API.

The Fremantle EDS fork also had a “freezable” (static) capability for libebook, which should probably be investigated. See, for instance: http://maemo.org/api_refs/5.0/5.0-final/libebook/EBookView.html#e-book-view-set-freezable

One possibility for adding the flag is to (ab)use the existing query language with a match for special fields:
* in the client: query = e_book_query_orv(e_book_query_field_exists("X-EDS-QUERY-FLAG-SKIP-DATA-DUMP"), <real query>, NULL);
* in the server: "if (query is OR-Query and first clause is exists("X-EDS-QUERY-FLAG-SKIP-DATA-DUMP")) { skip_data_dump = TRUE; query = second clause; }

[https://bugzilla.gnome.org/show_bug.cgi?id=652171 Upstream evolution-data-server bug]

=== Notify with UIDs Only ===

EBookView emits signals when contacts are changed, added, or removed. But those signals sends lists of entire EContacts, containing the entire contact data, though we often have all the data already and just want to know which contacts have changed.

[http://developer.gnome.org/libebook/stable/EBook.html#e-book-get-book-view e_book_get_book_view()] takes an [http://developer.gnome.org/libebook/stable/libebook-e-book-query.html#EBookQuery EBookQuery] which could be extended to take a new flag to make the EBookView send only the UID in the signals' EContacts. For instance, "X-EDS-QUERY-FLAG-NOTIFY-UID-ONLY". The UID is already stored separately for use as the primary key of the local-storage backend's BDB database, so no VCard parsing should be necessary to provide it.

This would allow easier integration with the QContact API, greatly improving performance.

Note that users of this feature should probably use just the base EVCard API on the provided EContact objects, to avoid extra processing and caching in the EContact code.

Note: We previously considered implementing e_book_get_book_view()'s requested fields parameter, but that changes the whole view, though we only really want to change what is sent for changed/new contacts.

Note: Chris discovered a dedicated e_book_get_contact_uids() API in the Fremantle EDS fork. The new API call was accepted upstream (in master) and [https://bugzilla.gnome.org/show_bug.cgi?id=651446#c11 backported as a patch] (used in the Meego package) for the gnome-2-32 branch, but this is not a full solution.

[https://bugzilla.gnome.org/show_bug.cgi?id=652172 Upstream evolution-data-server bug] (DONE)

== Optional Parsing of ebook Data in Client ==

The EDS client-side APIs currently parse and (temporarily) store the parsed details of VCards. However, this is often unnecessary because APIs such as e_book_get_book_view() cause it to return the whole VCard anyway. There will be some performance gain from delaying VCard parsing until it is really needed by calls to functions such as e_contact_new_from_vcard().

libebook provides different functionality:
# exchange data as vCard with EDS daemon
# parse items and wrap them in a C API

QtContacts replaces the second part. An obvious performance improvement is to only do the second part if the additional C APIs are really called. The following calls then transfer data without any additional parsing in libebook:
* e_book_get_contact()/asynchronous read/view + e_vcard_to_string()
* e_contact_new_from_vcard() + e_book_add_contact()/e_book_commit_contact()/asynchronous storing

Note that libecal (for iCalendar) can already do this, though libebook can't yet.

TODO:
# add delayed parsing to libebook, triggered by API call which needs parsed data

PROBLEM:
# If delayed parsing fails, how can error be reported?
# Should delayed parsing be done always (e_vcard_to_string())) or only if requested? How would that be configured by a libebook user?

An [http://build.meego.com/package/view_file?file=EBookView-Implement-delayed-vcard-parsing.patch&package=evolution-data-server&project=eds&srcmd5=977d2b885bed818cb616599749407e79 e_book_view_set_parse_vcards() patch], already used in the Meego package, partly implements this already. This patch was taken from an old fremantle EDS patch that was never actually used, and is generally considered to be a bad hack that should not be accepted upstream.

[https://bugzilla.gnome.org/show_bug.cgi?id=652173 Upstream evolution-data-server bug]

== Asynchronous API for storing items in libecal ==

libecal should allow the caller to request storing of data without blocking on processing of the data. It should notify the caller when the data has been stored, or when processing has failed.

These functions may require async versions:
* icalcomponent_new_from_string()
* e_cal_modify_object[_with_mod]
* e_cal_create_object()

Making libical asynchronous is almost impossible without going multithreaded. Let's focus on e_cal D-Bus communication first.

Rob started some work on this in the rbradford-wip-ecal-async-api branch. Perhaps you can reuse some of that work. It is not functional, but might be a source of inspiration or code fragments.

[https://bugzilla.gnome.org/show_bug.cgi?id=652174 Upstream evolution-data-server bug]

== More Efficient Access to Meta Data for Change Tracking ==

This was discussed a while ago: [[http://www.mail-archive.com/evolution-hackers@gnome.org/msg03029.html
[Evolution-hackers] concurrent modifications of items in GUI and EDS database]]

Data synchronization only needs to know which items exist (list of local IDs) and which revision of each item is currently stored. Revisions can be identified with a string which changes for each modification. Restoring an older revision should (but doesn't have to) restore the revision string.

For calendar, UID+RECURRENCE-ID and LAST-MODIFIED can be used. For contacts, UID + REV.

By comparing a stored list against the current one, changes can be detected without relying on journals in the daemon or comparison of time stamps. Journals are hard to maintain (how many of them, how long?). Comparison of time stamps has problems when changes are allowed while a sync runs and assumes that system time is linear (which is not always true).

As an additional use case, local ID + revision string could be used to prevent overwriting stale data in the server, as in HTTP PUT eTag checks.

TODO:

The Maemo Fremantle port of EDS, which is of course already open source, apparently has some solution for this, according to Mathias Hasselman, so that should be investigated first.

However, this should probably be made possible by:

# Extending the EDS query language with a flag that limits the delivered data to just:
## UID and REV for libebook (REV is in VCard 3.0)
## UID+RECURRENCE-ID+LAST-MODIFIED, for libecal
# Implementing these queries for the local contact and calendar backends. The query would be rejected, with an error, for other backends.

[https://bugzilla.gnome.org/show_bug.cgi?id=652179 libebook: Upstream evolution-data-server bug] 
[https://bugzilla.gnome.org/show_bug.cgi?id=652180 libecal: Upstream evolution-data-server bug]

== Atomic Updates ==

Later, implement atomic changes as discussed in the mail thread: [http://www.mail-archive.com/evolution-hackers@gnome.org/msg03029.html]

== Quick check for "data changed" ==

The libebook and libical APIs should offer a way to discover if any data whatsoever has changed. This would, for instance, help SyncEvolution to avoid unnecessary detailed checking. This could just check the modification dates of local storage files, such as the BDB .db file, or .ics file, as long as we do not use those files to store permanent caches of parsed data. This would be much quicker than listing items for the whole database and comparing their revisions strings, as is currently necessary.

This would require small API additions, to check that the opaque revision strings are unchanged, such as

gchar* e_book_get_revision(EBook *book)
gboolean e_book_check_revision(EBook *book , const gchar* revision)

and

gchar* e_cal_get_revision(ECal *ecal)
gboolean e_cal_check_revision(ECal ecal, const gchar revision)

The semantic would be this:
* If the revision string is empty or different, then the content of the database '''might''' have changed. A more detailed check is necessary to determine if it has really changed. This will be the case for non-local storage.
* If it is non-empty and the same as before, the nothing has changed. No further check will be necessary.

[https://bugzilla.gnome.org/show_bug.cgi?id=652175 libebook: Upstream evolution-data-server bug] 
[https://bugzilla.gnome.org/show_bug.cgi?id=652177 libecal: Upstream evolution-data-server bug]

== Contacts: Store PHOTO Data as Plain Files ==

Photos should be transferred over D-Bus via a filepath to a local file rather than transferring the actual photo data over D-Bus. The VCard format already allows this option (See the PHOTO property). We must be careful to keep these local files in sync and to remove them when contacts are removed, and to restrict access to the files from other processes.

There should be utility code available to libebook users to convert between the different representations.

Goals:
# Apps should be able to create photo files without ever passing the data to libebook.
# External PHOTO file and corresponding contact must be kept in sync:
## If a contact is deleted, the photo also needs to be removed.
## If a photo file is created and then storing the contact fails, who removes the file?
# Access to photo data must be limited to processes which have the right permissions.

Proposal:
* For each ~/.local/share/evolution/foo calendar, store photos in <photo path>=~/.local/share/evolution/foo/photos, with same permissions.
* New photo files are created with a [http://en.wikipedia.org/wiki/Universally_unique_identifier UUID] as base name and suffix according to the file content.
* If a contact is deleted and has a PHOTO;VALUE=uri:file://<photo path>/*, delete that file.
* For apps which want to create the photo file directly:
** Apps get a file descriptor and the path from a new libebook API.
** They own the file until the EDS server acknowledges the successful storing of a contact with a matching PHOTO;VALUE=uri:file property. If that fails, the app must remove the file.
* Add new API calls to libebook which split out resp. include data inline.
** Must track ownership of file after splitting it out and before storing - might never get stored!

Open:
* Should there be automatic garbage collection of stale photos in case something went wrong and a photo wasn't properly deleted?
* Trust TYPE=JPEG/PNG/... or look into file content to determine type?

[https://bugzilla.gnome.org/show_bug.cgi?id=652178 Upstream evolution-data-server bug]

== Future: Content Protection ==

Only authorized applications (= processes) with the necessary permissions should be able to read/write data. Exact permissions and mechanisms for verifying them to be defined as part of the (revised?) MeeGo architecture.

This can be added as follows:
# lock down file permissions
# add permission checks to EDS D-Bus servers

Currently not planned. Depends on the future MeeGo security infrastructure.

Maliit

2011-06-16T08:17:08Z

Murrayc: Rewrite introductory sentence.

[[Category:Maliit]][[Category:MeeGo Input Methods]]
{{MaliitNavigationBar}}

[[File:Maliit2.png|448px|link=]]

Maliit provides a flexible and cross-platform input method framework, including a virtual keyboard. It works on all MeeGo user experiences, and in other GNU/Linux distributions.

== Goals ==

Maliit aims to be ''the'' input method project for MeeGo and other GNU/Linux-based embedded/mobile platforms. To achieve this we should provide:

'''A solid input method framework'''

This includes support for traditional input methods such as hardware and virtual keyboards. More exotic input methods should also be possible, such as handwriting recognition and voice input, and special purpose input methods such as date/time pickers, tag clouds, etc.

'''A core set of input methods'''

The default set of input methods should demonstrate the core capabilities of the framework and provide a good user experience that device manufacturers and integrators can use easily.

We also aim to:

'''Empower third-party developers'''

Customizing existing input methods or developing powerful new input methods should be simple, regardless of whether it is for profit, research or fun.

'''Build a diverse community of contributors'''

Employees from different companies, freelancers, members of academia, hackers and volunteers should all be welcomed and encouraged to contribute to the project.

== Features ==

=== For end users ===
Following are the high level end-user features supported by Maliit:

* Multitouch virtual keyboard
** Landscape and portrait QWERTY with dedicated symbol views
** Support for multiple languages and scripts (e.g. latin, cyrillic, arabic, chinese)
*** See current layouts at [http://meego.gitorious.org/meegotouch/meegotouch-inputmethodkeyboard/trees/master/m-keyboard/layouts MeeGo gitorius]
** Number/phone number layouts for entering numbers/phone numbers
* Context sensitive and dynamic action key
** e.g. replacing enter icon with search icon and highlighting the key in search fields - and respective search key inactive when search field empty
* Context sensitive layouts
** e.g. replacing ',' key with '@' in e-mail address fields
* Simple interactions (as defined in [http://meego.com/developers/ui-design-guidelines/handset/meego-basics MeeGo Basics])
** Swipe sideways to easily switch between different active keyboard layouts and other input methods
** Swipe down to close virtual keyboard or alternatively tap outiside the active input area to close keyboard
* Low-latency haptics typing feedback (with feedback framework backend); sound,tactile

[[File:Text-input-1.png|Virtual keyboard interaction : opening and closing the keyboard]]

* Cut/Copy/Paste for text input
** Separate text editor widget for text input fields
* Error correction / word prediction for virtual keyboard (with error correction / prediction engine)
** Separate widget for correction/prediction candidates
* Hardware keyboard
** Long pressing of keys either to autorepeat (e.g. arrow keys, backspace) or to input secondary characters (e.g. numbers, symbols)
** Possibility to extend hardware keys with virtual keys (e.g. for additional symbols)

[[File:Text-input-2a.png|Text input with hardware keyboard]]

* Toolbar for appliction specific input content
** For placing buttons, labels etc.
** Toolbar located on top of virtual keyboard, with hardware keyboard in the bottom of the screen

[[File:Confirming.png|Placing application specific actions in the input toolbar, when application UI is not visible or "Enter" action is not adequate]]

* External input methods
** e.g. Bluetooth keyboards

=== For integrators and developers ===
* Input Methods can be implemented as plugins
* Error correction/prediction engines can be implemented as plugins
* Framework is licensed LGPL
* Keyboard has theming abilities via CSS file
* Meego Keyboard has customizable layout files
* Meego Feedback framework has swappable backends (for haptics et.c.)
* Support for additional toolkits can be done by using the DBus connection for the Input Context

== Maliit in action ==

Maliit is used on a number of devices and form factors, and can easily be adapted to run on new ones.

* [http://www.linpus.com/products_meego_slate.html Linpus Slate] ([http://www.linpus.com/images/slate/kb.png screenshot showing Linpus virtual keyboard])
* [http://appdeveloper.intel.com/en-us/meego-iso-esla Intel MeeGo Tablet] ([http://www.engadget.com/photos/new-meego-tablet-user-experience-hands-on-at-mwc-2011/#3877322 pictures showing Swype keyboard])
* [http://www.youtube.com/watch?v=l_7f3Lxd4_I GNOME3 on a WeTab (video)]

<gallery widths="250px" heights="250px" perrow="3">
Image:Lenovo_meego_vkb.jpg| [[Maliit/MeeGo 1.1#Netbook|MeeGo 1.1 Netbook]] with MeeGo Keyboard on Lenovo S10-3T Ideapad
Image:Meego1.1_handset_on_aava.jpg| [http://meego.com/downloads/releases/1.1/meego-v1.1-handset MeeGo 1.1 Handset] with MeeGo Keyboard on Aava Mobile
Image:tegra-vkb2.jpg| MeeGo 1.2 prerelease on [[ARM/TEGRA2|Advent Vega, Tegra2]]
Image:Maliit-on-lucid.png | Ubuntu Lucid (10.04)
Image:Maliit-in-GNOME3.jpg | GNOME3 on a WeTab
Image:Maliit-on-KDE-Plasma.jpg | KDE, using Plasma
[[File:TransparentPlaceholder.png|link=http://wikipedia.org|caption]]
</gallery>

== Communication channels ==

'''Mailing list:''' meego-inputmethods@lists.meego.com ([http://lists.meego.com/listinfo/meego-inputmethods listinfo] | [http://lists.meego.com/pipermail/meego-inputmethods/ archives])

'''Bugtracker:''' bugs.meego.com, Product: OS, Component: Virtual Keyboard ([https://bugs.meego.com/buglist.cgi?classification=MeeGo%20Platform&query_based_on=virtual%20keyboard&query_format=advanced&bug_status=NEW&bug_status=NEEDINFO&bug_status=INDEFINITION&bug_status=ASSIGNED&bug_status=ACCEPTED&bug_status=WAITING%20FOR%20UPSTREAM&bug_status=WAITING&bug_status=REOPENED&component=Virtual%20keyboard&product=OS%20Base&known_name=virtual%20keyboard open bugs] |
[https://bugs.meego.com/enter_bug.cgi?product=OS%20Base&component=Virtual%20keyboard file new bug])

'''IRC:''' Visit us in #meego-inputmethods at freenode.net ([http://webchat.freenode.net/?randomnick=1&channels=meego-inputmethods&uio=d4 webchat])

== News ==
* 07.06.2011 - Session at MeeGo Spring Conference 2011. [http://sf2011.meego.com/program/sessions/developing-custom-input-methods-meego Info, Recording]
* 08.04.2011 - MeeGo Keyboard is relicensed with BSD license: [http://lists.meego.com/pipermail/meego-dev/2011-April/482578.html Announcement]
* 29.03.2011 - Held first offical IRC meeting: [[Maliit/Meetings|How can we simplify development and maintenance of input contexts?]]
* 04.03.2011 - API/ABI break in 0.20.0: [http://lists.meego.com/pipermail/meego-dev/2011-March/481869.html meego-im-framework is removing dependency to libmeegotouch from its API]
* 23.02.2011 - Packages available for Ubuntu Lucid/Maverick to ease development. [http://taschenorakel.de/michael/2011/02/24/meego-input-methods-your-desktop/ Blogpost], [[#Packages for Ubuntu .2810.04 Lucid Lynx .2F 10.10 Maveric Meerkat.29|documentation]]
* 17.02.2011 - Claudio Saavedra started on MeeGo+Gtk integration work, focusing on input methods. [http://people.gnome.org/~csaavedra/news-2011-02.html#D10 Blogpost]
* 13.02.2011 - MeeGo Tablet UX demoed at Mobile World Conference, with Swype keyboard. [http://www.engadget.com/photos/new-meego-tablet-user-experience-hands-on-at-mwc-2011/#3877322 Pictures]
* 01.02.2011 - API/ABI break in 0.19.41: [http://lists.meego.com/pipermail/meego-dev/2011-February/481353.html meegoimframework interfaces related to orientation change will be changed]
* 15.11.2010 - Architect and Lead Developer of Maliit give talk at MeeGo Conference. [http://conference2010.meego.com/session/meego-touch-input-method-frameworks Info, Recording]
* 15.11.2010 - Linpus shows off MeeGo based slate at MeeGo Conference, with keyboard based on Maliit.
* 30.06.2010 - MeeGo Handset UX Day 1 release, first public release of MeeGo Input Methods. [http://meego.com/community/blogs/valhalla/2010/meego-handset-project-day-1-here Announcement]

== Roadmap ==
See [[Maliit/Roadmap]]

== Upcoming events ==
* June 24th, MeeGo Freeday workshop, details TBA.
* August 6th-12th - Desktop Summit Berlin.

== Ideas ==
See [[Maliit/Ideas]]

== Developers ==
See [[Maliit/Development]]

== Documentation ==
See [[Maliit/Documentation]]

{{MaliitNavigationBar}}

Maliit

2011-06-16T08:14:27Z

Murrayc: Goals: Slight rewrite, though I think the overview needs to be more visual in general.

[[Category:Maliit]][[Category:MeeGo Input Methods]]
{{MaliitNavigationBar}}

[[File:Maliit2.png|448px|link=]]

Maliit provides a flexible and cross platform input method framework. It is usable on all MeeGo user experiences, and in other GNU/Linux distributions as well.

== Goals ==

Maliit aims to be ''the'' input method project for MeeGo and other GNU/Linux-based embedded/mobile platforms. To achieve this we should provide:

'''A solid input method framework'''

This includes support for traditional input methods such as hardware and virtual keyboards. More exotic input methods should also be possible, such as handwriting recognition and voice input, and special purpose input methods such as date/time pickers, tag clouds, etc.

'''A core set of input methods'''

The default set of input methods should demonstrate the core capabilities of the framework and provide a good user experience that device manufacturers and integrators can use easily.

We also aim to:

'''Empower third-party developers'''

Customizing existing input methods or developing powerful new input methods should be simple, regardless of whether it is for profit, research or fun.

'''Build a diverse community of contributors'''

Employees from different companies, freelancers, members of academia, hackers and volunteers should all be welcomed and encouraged to contribute to the project.

== Features ==

=== For end users ===
Following are the high level end-user features supported by Maliit:

* Multitouch virtual keyboard
** Landscape and portrait QWERTY with dedicated symbol views
** Support for multiple languages and scripts (e.g. latin, cyrillic, arabic, chinese)
*** See current layouts at [http://meego.gitorious.org/meegotouch/meegotouch-inputmethodkeyboard/trees/master/m-keyboard/layouts MeeGo gitorius]
** Number/phone number layouts for entering numbers/phone numbers
* Context sensitive and dynamic action key
** e.g. replacing enter icon with search icon and highlighting the key in search fields - and respective search key inactive when search field empty
* Context sensitive layouts
** e.g. replacing ',' key with '@' in e-mail address fields
* Simple interactions (as defined in [http://meego.com/developers/ui-design-guidelines/handset/meego-basics MeeGo Basics])
** Swipe sideways to easily switch between different active keyboard layouts and other input methods
** Swipe down to close virtual keyboard or alternatively tap outiside the active input area to close keyboard
* Low-latency haptics typing feedback (with feedback framework backend); sound,tactile

[[File:Text-input-1.png|Virtual keyboard interaction : opening and closing the keyboard]]

* Cut/Copy/Paste for text input
** Separate text editor widget for text input fields
* Error correction / word prediction for virtual keyboard (with error correction / prediction engine)
** Separate widget for correction/prediction candidates
* Hardware keyboard
** Long pressing of keys either to autorepeat (e.g. arrow keys, backspace) or to input secondary characters (e.g. numbers, symbols)
** Possibility to extend hardware keys with virtual keys (e.g. for additional symbols)

[[File:Text-input-2a.png|Text input with hardware keyboard]]

* Toolbar for appliction specific input content
** For placing buttons, labels etc.
** Toolbar located on top of virtual keyboard, with hardware keyboard in the bottom of the screen

[[File:Confirming.png|Placing application specific actions in the input toolbar, when application UI is not visible or "Enter" action is not adequate]]

* External input methods
** e.g. Bluetooth keyboards

=== For integrators and developers ===
* Input Methods can be implemented as plugins
* Error correction/prediction engines can be implemented as plugins
* Framework is licensed LGPL
* Keyboard has theming abilities via CSS file
* Meego Keyboard has customizable layout files
* Meego Feedback framework has swappable backends (for haptics et.c.)
* Support for additional toolkits can be done by using the DBus connection for the Input Context

== Maliit in action ==

Maliit is used on a number of devices and form factors, and can easily be adapted to run on new ones.

* [http://www.linpus.com/products_meego_slate.html Linpus Slate] ([http://www.linpus.com/images/slate/kb.png screenshot showing Linpus virtual keyboard])
* [http://appdeveloper.intel.com/en-us/meego-iso-esla Intel MeeGo Tablet] ([http://www.engadget.com/photos/new-meego-tablet-user-experience-hands-on-at-mwc-2011/#3877322 pictures showing Swype keyboard])
* [http://www.youtube.com/watch?v=l_7f3Lxd4_I GNOME3 on a WeTab (video)]

<gallery widths="250px" heights="250px" perrow="3">
Image:Lenovo_meego_vkb.jpg| [[Maliit/MeeGo 1.1#Netbook|MeeGo 1.1 Netbook]] with MeeGo Keyboard on Lenovo S10-3T Ideapad
Image:Meego1.1_handset_on_aava.jpg| [http://meego.com/downloads/releases/1.1/meego-v1.1-handset MeeGo 1.1 Handset] with MeeGo Keyboard on Aava Mobile
Image:tegra-vkb2.jpg| MeeGo 1.2 prerelease on [[ARM/TEGRA2|Advent Vega, Tegra2]]
Image:Maliit-on-lucid.png | Ubuntu Lucid (10.04)
Image:Maliit-in-GNOME3.jpg | GNOME3 on a WeTab
Image:Maliit-on-KDE-Plasma.jpg | KDE, using Plasma
[[File:TransparentPlaceholder.png|link=http://wikipedia.org|caption]]
</gallery>

== Communication channels ==

'''Mailing list:''' meego-inputmethods@lists.meego.com ([http://lists.meego.com/listinfo/meego-inputmethods listinfo] | [http://lists.meego.com/pipermail/meego-inputmethods/ archives])

'''Bugtracker:''' bugs.meego.com, Product: OS, Component: Virtual Keyboard ([https://bugs.meego.com/buglist.cgi?classification=MeeGo%20Platform&query_based_on=virtual%20keyboard&query_format=advanced&bug_status=NEW&bug_status=NEEDINFO&bug_status=INDEFINITION&bug_status=ASSIGNED&bug_status=ACCEPTED&bug_status=WAITING%20FOR%20UPSTREAM&bug_status=WAITING&bug_status=REOPENED&component=Virtual%20keyboard&product=OS%20Base&known_name=virtual%20keyboard open bugs] |
[https://bugs.meego.com/enter_bug.cgi?product=OS%20Base&component=Virtual%20keyboard file new bug])

'''IRC:''' Visit us in #meego-inputmethods at freenode.net ([http://webchat.freenode.net/?randomnick=1&channels=meego-inputmethods&uio=d4 webchat])

== News ==
* 07.06.2011 - Session at MeeGo Spring Conference 2011. [http://sf2011.meego.com/program/sessions/developing-custom-input-methods-meego Info, Recording]
* 08.04.2011 - MeeGo Keyboard is relicensed with BSD license: [http://lists.meego.com/pipermail/meego-dev/2011-April/482578.html Announcement]
* 29.03.2011 - Held first offical IRC meeting: [[Maliit/Meetings|How can we simplify development and maintenance of input contexts?]]
* 04.03.2011 - API/ABI break in 0.20.0: [http://lists.meego.com/pipermail/meego-dev/2011-March/481869.html meego-im-framework is removing dependency to libmeegotouch from its API]
* 23.02.2011 - Packages available for Ubuntu Lucid/Maverick to ease development. [http://taschenorakel.de/michael/2011/02/24/meego-input-methods-your-desktop/ Blogpost], [[#Packages for Ubuntu .2810.04 Lucid Lynx .2F 10.10 Maveric Meerkat.29|documentation]]
* 17.02.2011 - Claudio Saavedra started on MeeGo+Gtk integration work, focusing on input methods. [http://people.gnome.org/~csaavedra/news-2011-02.html#D10 Blogpost]
* 13.02.2011 - MeeGo Tablet UX demoed at Mobile World Conference, with Swype keyboard. [http://www.engadget.com/photos/new-meego-tablet-user-experience-hands-on-at-mwc-2011/#3877322 Pictures]
* 01.02.2011 - API/ABI break in 0.19.41: [http://lists.meego.com/pipermail/meego-dev/2011-February/481353.html meegoimframework interfaces related to orientation change will be changed]
* 15.11.2010 - Architect and Lead Developer of Maliit give talk at MeeGo Conference. [http://conference2010.meego.com/session/meego-touch-input-method-frameworks Info, Recording]
* 15.11.2010 - Linpus shows off MeeGo based slate at MeeGo Conference, with keyboard based on Maliit.
* 30.06.2010 - MeeGo Handset UX Day 1 release, first public release of MeeGo Input Methods. [http://meego.com/community/blogs/valhalla/2010/meego-handset-project-day-1-here Announcement]

== Roadmap ==
See [[Maliit/Roadmap]]

== Upcoming events ==
* June 24th, MeeGo Freeday workshop, details TBA.
* August 6th-12th - Desktop Summit Berlin.

== Ideas ==
See [[Maliit/Ideas]]

== Developers ==
See [[Maliit/Development]]

== Documentation ==
See [[Maliit/Documentation]]

{{MaliitNavigationBar}}

Maliit

2011-06-16T08:09:15Z

Murrayc: Move actual overview text to the top of the overview

[[Category:Maliit]][[Category:MeeGo Input Methods]]
{{MaliitNavigationBar}}

[[File:Maliit2.png|448px|link=]]

Maliit provides a flexible and cross platform input method framework. It is usable on all MeeGo user experiences, and in other GNU/Linux distributions as well.

== Goals ==

The overall goal of Maliit is to be ''the'' input method project for MeeGo and other GNU/Linux-based embedded/mobile platforms. To achieve this we aim to:

'''Provide a solid input method framework'''

This includes explicit support for traditional input methods like hardware and virtual keyboards. More exotic input methods like handwriting recognition and voice input, and special purpose input methods like date/time pickers, tag clouds, et.c. should also be possible.

'''Provide a core set of input methods'''

The default set of input methods should demonstrate the core capabilites of the framework, and provide a good user experience that device manufacturers and integrators can use with little to no adaptation.

'''Empower third-party developers'''

Customizing existing or developing powerful new input methods should be simple, regardless of whether it is for profit, research or fun.

'''Have a diverse community of contributors'''

Employees from different companies, freelancers, members of academia, hackers and volunteers should all be welcomed and encouraged to contribute to the project.

== Features ==

=== For end users ===
Following are the high level end-user features supported by Maliit:

* Multitouch virtual keyboard
** Landscape and portrait QWERTY with dedicated symbol views
** Support for multiple languages and scripts (e.g. latin, cyrillic, arabic, chinese)
*** See current layouts at [http://meego.gitorious.org/meegotouch/meegotouch-inputmethodkeyboard/trees/master/m-keyboard/layouts MeeGo gitorius]
** Number/phone number layouts for entering numbers/phone numbers
* Context sensitive and dynamic action key
** e.g. replacing enter icon with search icon and highlighting the key in search fields - and respective search key inactive when search field empty
* Context sensitive layouts
** e.g. replacing ',' key with '@' in e-mail address fields
* Simple interactions (as defined in [http://meego.com/developers/ui-design-guidelines/handset/meego-basics MeeGo Basics])
** Swipe sideways to easily switch between different active keyboard layouts and other input methods
** Swipe down to close virtual keyboard or alternatively tap outiside the active input area to close keyboard
* Low-latency haptics typing feedback (with feedback framework backend); sound,tactile

[[File:Text-input-1.png|Virtual keyboard interaction : opening and closing the keyboard]]

* Cut/Copy/Paste for text input
** Separate text editor widget for text input fields
* Error correction / word prediction for virtual keyboard (with error correction / prediction engine)
** Separate widget for correction/prediction candidates
* Hardware keyboard
** Long pressing of keys either to autorepeat (e.g. arrow keys, backspace) or to input secondary characters (e.g. numbers, symbols)
** Possibility to extend hardware keys with virtual keys (e.g. for additional symbols)

[[File:Text-input-2a.png|Text input with hardware keyboard]]

* Toolbar for appliction specific input content
** For placing buttons, labels etc.
** Toolbar located on top of virtual keyboard, with hardware keyboard in the bottom of the screen

[[File:Confirming.png|Placing application specific actions in the input toolbar, when application UI is not visible or "Enter" action is not adequate]]

* External input methods
** e.g. Bluetooth keyboards

=== For integrators and developers ===
* Input Methods can be implemented as plugins
* Error correction/prediction engines can be implemented as plugins
* Framework is licensed LGPL
* Keyboard has theming abilities via CSS file
* Meego Keyboard has customizable layout files
* Meego Feedback framework has swappable backends (for haptics et.c.)
* Support for additional toolkits can be done by using the DBus connection for the Input Context

== Maliit in action ==

Maliit is used on a number of devices and form factors, and can easily be adapted to run on new ones.

* [http://www.linpus.com/products_meego_slate.html Linpus Slate] ([http://www.linpus.com/images/slate/kb.png screenshot showing Linpus virtual keyboard])
* [http://appdeveloper.intel.com/en-us/meego-iso-esla Intel MeeGo Tablet] ([http://www.engadget.com/photos/new-meego-tablet-user-experience-hands-on-at-mwc-2011/#3877322 pictures showing Swype keyboard])
* [http://www.youtube.com/watch?v=l_7f3Lxd4_I GNOME3 on a WeTab (video)]

<gallery widths="250px" heights="250px" perrow="3">
Image:Lenovo_meego_vkb.jpg| [[Maliit/MeeGo 1.1#Netbook|MeeGo 1.1 Netbook]] with MeeGo Keyboard on Lenovo S10-3T Ideapad
Image:Meego1.1_handset_on_aava.jpg| [http://meego.com/downloads/releases/1.1/meego-v1.1-handset MeeGo 1.1 Handset] with MeeGo Keyboard on Aava Mobile
Image:tegra-vkb2.jpg| MeeGo 1.2 prerelease on [[ARM/TEGRA2|Advent Vega, Tegra2]]
Image:Maliit-on-lucid.png | Ubuntu Lucid (10.04)
Image:Maliit-in-GNOME3.jpg | GNOME3 on a WeTab
Image:Maliit-on-KDE-Plasma.jpg | KDE, using Plasma
[[File:TransparentPlaceholder.png|link=http://wikipedia.org|caption]]
</gallery>

== Communication channels ==

'''Mailing list:''' meego-inputmethods@lists.meego.com ([http://lists.meego.com/listinfo/meego-inputmethods listinfo] | [http://lists.meego.com/pipermail/meego-inputmethods/ archives])

'''Bugtracker:''' bugs.meego.com, Product: OS, Component: Virtual Keyboard ([https://bugs.meego.com/buglist.cgi?classification=MeeGo%20Platform&query_based_on=virtual%20keyboard&query_format=advanced&bug_status=NEW&bug_status=NEEDINFO&bug_status=INDEFINITION&bug_status=ASSIGNED&bug_status=ACCEPTED&bug_status=WAITING%20FOR%20UPSTREAM&bug_status=WAITING&bug_status=REOPENED&component=Virtual%20keyboard&product=OS%20Base&known_name=virtual%20keyboard open bugs] |
[https://bugs.meego.com/enter_bug.cgi?product=OS%20Base&component=Virtual%20keyboard file new bug])

'''IRC:''' Visit us in #meego-inputmethods at freenode.net ([http://webchat.freenode.net/?randomnick=1&channels=meego-inputmethods&uio=d4 webchat])

== News ==
* 07.06.2011 - Session at MeeGo Spring Conference 2011. [http://sf2011.meego.com/program/sessions/developing-custom-input-methods-meego Info, Recording]
* 08.04.2011 - MeeGo Keyboard is relicensed with BSD license: [http://lists.meego.com/pipermail/meego-dev/2011-April/482578.html Announcement]
* 29.03.2011 - Held first offical IRC meeting: [[Maliit/Meetings|How can we simplify development and maintenance of input contexts?]]
* 04.03.2011 - API/ABI break in 0.20.0: [http://lists.meego.com/pipermail/meego-dev/2011-March/481869.html meego-im-framework is removing dependency to libmeegotouch from its API]
* 23.02.2011 - Packages available for Ubuntu Lucid/Maverick to ease development. [http://taschenorakel.de/michael/2011/02/24/meego-input-methods-your-desktop/ Blogpost], [[#Packages for Ubuntu .2810.04 Lucid Lynx .2F 10.10 Maveric Meerkat.29|documentation]]
* 17.02.2011 - Claudio Saavedra started on MeeGo+Gtk integration work, focusing on input methods. [http://people.gnome.org/~csaavedra/news-2011-02.html#D10 Blogpost]
* 13.02.2011 - MeeGo Tablet UX demoed at Mobile World Conference, with Swype keyboard. [http://www.engadget.com/photos/new-meego-tablet-user-experience-hands-on-at-mwc-2011/#3877322 Pictures]
* 01.02.2011 - API/ABI break in 0.19.41: [http://lists.meego.com/pipermail/meego-dev/2011-February/481353.html meegoimframework interfaces related to orientation change will be changed]
* 15.11.2010 - Architect and Lead Developer of Maliit give talk at MeeGo Conference. [http://conference2010.meego.com/session/meego-touch-input-method-frameworks Info, Recording]
* 15.11.2010 - Linpus shows off MeeGo based slate at MeeGo Conference, with keyboard based on Maliit.
* 30.06.2010 - MeeGo Handset UX Day 1 release, first public release of MeeGo Input Methods. [http://meego.com/community/blogs/valhalla/2010/meego-handset-project-day-1-here Announcement]

== Roadmap ==
See [[Maliit/Roadmap]]

== Upcoming events ==
* June 24th, MeeGo Freeday workshop, details TBA.
* August 6th-12th - Desktop Summit Berlin.

== Ideas ==
See [[Maliit/Ideas]]

== Developers ==
See [[Maliit/Development]]

== Documentation ==
See [[Maliit/Documentation]]

{{MaliitNavigationBar}}

Architecture/planning/evolution-data-server/eds-improvements

2011-06-14T10:38:42Z

Murrayc: Fix typo, though this will break links to this sub-section.

This page describes improvements that are desirable or even essential for good performance of Evolution Data Server (EDS) as PIM storage in MeeGo.

All of these changes should be developed for the evolution-data-server gnome-2-32 branch, as used by MeeGo 1.2, then ported to master for review by upstream maintainers. Most changes will not be accepted in upstream's gnome-2-32 branch, because is is stable, but Meego will apply them as patches to its package.

See the list of [https://bugzilla.gnome.org/buglist.cgi?status_whiteboard_type=allwordssubstr;query_format=advanced;field0-0-0=product;status_whiteboard=meego;bug_status=UNCONFIRMED;bug_status=NEW;bug_status=ASSIGNED;bug_status=REOPENED;bug_status=NEEDINFO;type0-0-0=substring;value0-0-0=Evolution-Data-Server "meego" bugs in the upstream evolution-data-server bugzilla].

== Change Tracking ==

EDS is based on the model/view/controller principle: an application opens a database (calendar or addressbook) and requests a view with server-side filtering, then the application is sent:
# all data covered by the view
# the "all data sent" signal
# changes (add/update/remove) while the view exists

Applications modify data with asynchronous requests (controller) and keep showing the old data until the change notification comes in. The exact changes made to the data are determined by the server (for example when meeting invitations are sent).

QtContacts and KCalCore do not have such a strict model. With QtContacts the app simply loads and stores contacts and is responsible for maintaining its own view. Changes are sent for changes made after the database was opened. KCalCore is based on the concept of loading a calendar (partially) into the client, modifying it there, then writing it back. Changes are also sent while the storage is open.

TODO:
* https://bugs.meego.com/show_bug.cgi?id=18760 - "QtContacts-EDS + EDS: more flexible EBookView (UID only, only future changes)"

=== Suppress Unnecessary Data Transfer ===

EDS currently sends all data when a view is requested, which is unnecessary when the client is only interested in change notifications. An extension to the EDS query language could optionally suppresses that sending of existing data when the view is opened, to be implemented only for the local contacts and calendar backends. This would allow easier integration with the QContact API.

The Fremantle EDS fork also had a “freezable” (static) capability for libebook, which should probably be investigated. See, for instance: http://maemo.org/api_refs/5.0/5.0-final/libebook/EBookView.html#e-book-view-set-freezable

One possibility for adding the flag is to (ab)use the existing query language with a match for special fields:
* in the client: query = e_book_query_orv(e_book_query_field_exists("X-EDS-QUERY-FLAG-SKIP-DATA-DUMP"), <real query>, NULL);
* in the server: "if (query is OR-Query and first clause is exists("X-EDS-QUERY-FLAG-SKIP-DATA-DUMP")) { skip_data_dump = TRUE; query = second clause; }

[https://bugzilla.gnome.org/show_bug.cgi?id=652171 Upstream evolution-data-server bug]

=== Notify with UIDs Only ===

EBookView emits signals when contacts are changed, added, or removed. But those signals sends lists of entire EContacts, containing the entire contact data, though we often have all the data already and just want to know which contacts have changed.

[http://developer.gnome.org/libebook/stable/EBook.html#e-book-get-book-view e_book_get_book_view()] takes an [http://developer.gnome.org/libebook/stable/libebook-e-book-query.html#EBookQuery EBookQuery] which could be extended to take a new flag to make the EBookView send only the UID in the signals' EContacts. For instance, "X-EDS-QUERY-FLAG-NOTIFY-UID-ONLY". The UID is already stored separately for use as the primary key of the local-storage backend's BDB database, so no VCard parsing should be necessary to provide it.

This would allow easier integration with the QContact API, greatly improving performance.

Note that users of this feature should probably use just the base EVCard API on the provided EContact objects, to avoid extra processing and caching in the EContact code.

Note: We previously considered implementing e_book_get_book_view()'s requested fields parameter, but that changes the whole view, though we only really want to change what is sent for changed/new contacts.

Note: Chris discovered a dedicated e_book_get_contact_uids() API in the Fremantle EDS fork. The new API call was accepted upstream (in master) and [https://bugzilla.gnome.org/show_bug.cgi?id=651446#c11 backported as a patch] (used in the Meego package) for the gnome-2-32 branch, but this is not a full solution.

[https://bugzilla.gnome.org/show_bug.cgi?id=652172 Upstream evolution-data-server bug]

== Optional Parsing of ebook Data in Client ==

The EDS client-side APIs currently parse and (temporarily) store the parsed details of VCards. However, this is often unnecessary because APIs such as e_book_get_book_view() cause it to return the whole VCard anyway. There will be some performance gain from delaying VCard parsing until it is really needed by calls to functions such as e_contact_new_from_vcard().

libebook provides different functionality:
# exchange data as vCard with EDS daemon
# parse items and wrap them in a C API

QtContacts replaces the second part. An obvious performance improvement is to only do the second part if the additional C APIs are really called. The following calls then transfer data without any additional parsing in libebook:
* e_book_get_contact()/asynchronous read/view + e_vcard_to_string()
* e_contact_new_from_vcard() + e_book_add_contact()/e_book_commit_contact()/asynchronous storing

Note that libecal (for iCalendar) can already do this, though libebook can't yet.

TODO:
# add delayed parsing to libebook, triggered by API call which needs parsed data

PROBLEM:
# If delayed parsing fails, how can error be reported?
# Should delayed parsing be done always (e_vcard_to_string())) or only if requested? How would that be configured by a libebook user?

An [http://build.meego.com/package/view_file?file=EBookView-Implement-delayed-vcard-parsing.patch&package=evolution-data-server&project=eds&srcmd5=977d2b885bed818cb616599749407e79 e_book_view_set_parse_vcards() patch], already used in the Meego package, partly implements this already. This patch was taken from an old fremantle EDS patch that was never actually used, and is generally considered to be a bad hack that should not be accepted upstream.

[https://bugzilla.gnome.org/show_bug.cgi?id=652173 Upstream evolution-data-server bug]

== Asynchronous API for storing items in libecal ==

libecal should allow the caller to request storing of data without blocking on processing of the data. It should notify the caller when the data has been stored, or when processing has failed.

These functions may require async versions:
* icalcomponent_new_from_string()
* e_cal_modify_object[_with_mod]
* e_cal_create_object()

Making libical asynchronous is almost impossible without going multithreaded. Let's focus on e_cal D-Bus communication first.

Rob started some work on this in the rbradford-wip-ecal-async-api branch. Perhaps you can reuse some of that work. It is not functional, but might be a source of inspiration or code fragments.

[https://bugzilla.gnome.org/show_bug.cgi?id=652174 Upstream evolution-data-server bug]

== More Efficient Access to Meta Data for Change Tracking ==

This was discussed a while ago: [[http://www.mail-archive.com/evolution-hackers@gnome.org/msg03029.html
[Evolution-hackers] concurrent modifications of items in GUI and EDS database]]

Data synchronization only needs to know which items exist (list of local IDs) and which revision of each item is currently stored. Revisions can be identified with a string which changes for each modification. Restoring an older revision should (but doesn't have to) restore the revision string.

For calendar, UID+RECURRENCE-ID and LAST-MODIFIED can be used. For contacts, UID + REV.

By comparing a stored list against the current one, changes can be detected without relying on journals in the daemon or comparison of time stamps. Journals are hard to maintain (how many of them, how long?). Comparison of time stamps has problems when changes are allowed while a sync runs and assumes that system time is linear (which is not always true).

As an additional use case, local ID + revision string could be used to prevent overwriting stale data in the server, as in HTTP PUT eTag checks.

TODO:

The Maemo Fremantle port of EDS, which is of course already open source, apparently has some solution for this, according to Mathias Hasselman, so that should be investigated first.

However, this should probably be made possible by:

# Extending the EDS query language with a flag that limits the delivered data to just:
## UID and REV for libebook (REV is in VCard 3.0)
## UID+RECURRENCE-ID+LAST-MODIFIED, for libecal
# Implementing these queries for the local contact and calendar backends. The query would be rejected, with an error, for other backends.

[https://bugzilla.gnome.org/show_bug.cgi?id=652179 libebook: Upstream evolution-data-server bug] 
[https://bugzilla.gnome.org/show_bug.cgi?id=652180 libecal: Upstream evolution-data-server bug]

== Atomic Updates ==

Later, implement atomic changes as discussed in the mail thread: [http://www.mail-archive.com/evolution-hackers@gnome.org/msg03029.html]

== Quick check for "data changed" ==

The libebook and libical APIs should offer a way to discover if any data whatsoever has changed. This would, for instance, help SyncEvolution to avoid unnecessary detailed checking. This could just check the modification dates of local storage files, such as the BDB .db file, or .ics file, as long as we do not use those files to store permanent caches of parsed data. This would be much quicker than listing items for the whole database and comparing their revisions strings, as is currently necessary.

This would require small API additions, to check that the opaque revision strings are unchanged, such as

gchar* e_book_get_revision(EBook *book)
gboolean e_book_check_revision(EBook *book , const gchar* revision)

and

gchar* e_cal_get_revision(ECal *ecal)
gboolean e_cal_check_revision(ECal ecal, const gchar revision)

The semantic would be this:
* If the revision string is empty or different, then the content of the database '''might''' have changed. A more detailed check is necessary to determine if it has really changed. This will be the case for non-local storage.
* If it is non-empty and the same as before, the nothing has changed. No further check will be necessary.

[https://bugzilla.gnome.org/show_bug.cgi?id=652175 libebook: Upstream evolution-data-server bug] 
[https://bugzilla.gnome.org/show_bug.cgi?id=652177 libecal: Upstream evolution-data-server bug]

== Contacts: Store PHOTO Data as Plain Files ==

Photos should be transferred over D-Bus via a filepath to a local file rather than transferring the actual photo data over D-Bus. The VCard format already allows this option (See the PHOTO property). We must be careful to keep these local files in sync and to remove them when contacts are removed, and to restrict access to the files from other processes.

There should be utility code available to libebook users to convert between the different representations.

Goals:
# Apps should be able to create photo files without ever passing the data to libebook.
# External PHOTO file and corresponding contact must be kept in sync:
## If a contact is deleted, the photo also needs to be removed.
## If a photo file is created and then storing the contact fails, who removes the file?
# Access to photo data must be limited to processes which have the right permissions.

Proposal:
* For each ~/.local/share/evolution/foo calendar, store photos in <photo path>=~/.local/share/evolution/foo/photos, with same permissions.
* New photo files are created with a [http://en.wikipedia.org/wiki/Universally_unique_identifier UUID] as base name and suffix according to the file content.
* If a contact is deleted and has a PHOTO;VALUE=uri:file://<photo path>/*, delete that file.
* For apps which want to create the photo file directly:
** Apps get a file descriptor and the path from a new libebook API.
** They own the file until the EDS server acknowledges the successful storing of a contact with a matching PHOTO;VALUE=uri:file property. If that fails, the app must remove the file.
* Add new API calls to libebook which split out resp. include data inline.
** Must track ownership of file after splitting it out and before storing - might never get stored!

Open:
* Should there be automatic garbage collection of stale photos in case something went wrong and a photo wasn't properly deleted?
* Trust TYPE=JPEG/PNG/... or look into file content to determine type?

[https://bugzilla.gnome.org/show_bug.cgi?id=652178 Upstream evolution-data-server bug]

== Future: Content Protection ==

Only authorized applications (= processes) with the necessary permissions should be able to read/write data. Exact permissions and mechanisms for verifying them to be defined as part of the (revised?) MeeGo architecture.

This can be added as follows:
# lock down file permissions
# add permission checks to EDS D-Bus servers

Currently not planned. Depends on the future MeeGo security infrastructure.

Architecture/planning/evolution-data-server/eds-improvements

2011-06-09T10:24:49Z

Murrayc: Add a link to all upstream e-d-s bugs for meego.

This page describes improvements that are desirable or even essential for good performance of Evolution Data Server (EDS) as PIM storage in MeeGo.

All of these changes should be developed for the evolution-data-server gnome-2-32 branch, as used by MeeGo 1.2, then ported to master for review by upstream maintainers. Most changes will not be accepted in upstream's gnome-2-32 branch, because is is stable, but Meego will apply them as patches to its package.

See the list of [https://bugzilla.gnome.org/buglist.cgi?status_whiteboard_type=allwordssubstr;query_format=advanced;field0-0-0=product;status_whiteboard=meego;bug_status=UNCONFIRMED;bug_status=NEW;bug_status=ASSIGNED;bug_status=REOPENED;bug_status=NEEDINFO;type0-0-0=substring;value0-0-0=Evolution-Data-Server "meego" bugs in the upstream evolution-data-server bugzilla].

== Change Tracking ==

EDS is based on the model/view/controller principle: an application opens a database (calendar or addressbook) and requests a view with server-side filtering, then the application is sent:
# all data covered by the view
# the "all data sent" signal
# changes (add/update/remove) while the view exists

Applications modify data with asynchronous requests (controller) and keep showing the old data until the change notification comes in. The exact changes made to the data are determined by the server (for example when meeting invitations are sent).

QtContacts and KCalCore do not have such a strict model. With QtContacts the app simply loads and stores contacts and is responsible for maintaining its own view. Changes are sent for changes made after the database was opened. KCalCore is based on the concept of loading a calendar (partially) into the client, modifying it there, then writing it back. Changes are also sent while the storage is open.

TODO:
* https://bugs.meego.com/show_bug.cgi?id=18760 - "QtContacts-EDS + EDS: more flexible EBookView (UID only, only future changes)"

=== Suppress Unnecessary Data Transfer ===

EDS currently sends all data when a view is requested, which is unnecessary when the client is only interested in change notifications. An extension to the EDS query language could optionally suppresses that sending of existing data when the view is opened, to be implemented only for the local contacts and calendar backends. This would allow easier integration with the QContact API.

The Fremantle EDS fork also had a “freezable” (static) capability for libebook, which should probably be investigated. See, for instance: http://maemo.org/api_refs/5.0/5.0-final/libebook/EBookView.html#e-book-view-set-freezable

One possibility for adding the flag is to (ab)use the existing query language with a match for special fields:
* in the client: query = e_book_query_orv(e_book_query_field_exists("X-EDS-QUERY-FLAG-SKIP-DATA-DUMP"), <real query>, NULL);
* in the server: "if (query is OR-Query and first clause is exists("X-EDS-QUERY-FLAG-SKIP-DATA-DUMP")) { skip_data_dump = TRUE; query = second clause; }

[https://bugzilla.gnome.org/show_bug.cgi?id=652171 Upstream evolution-data-server bug]

=== Nofify with UIDs Only ===

EBookView emits signals when contacts are changed, added, or removed. But those signals sends lists of entire EContacts, containing the entire contact data, though we often have all the data already and just want to know which contacts have changed.

[http://developer.gnome.org/libebook/stable/EBook.html#e-book-get-book-view e_book_get_book_view()] takes an [http://developer.gnome.org/libebook/stable/libebook-e-book-query.html#EBookQuery EBookQuery] which could be extended to take a new flag to make the EBookView send only the UID in the signals' EContacts. For instance, "X-EDS-QUERY-FLAG-NOTIFY-UID-ONLY". The UID is already stored separately for use as the primary key of the local-storage backend's BDB database, so no VCard parsing should be necessary to provide it.

This would allow easier integration with the QContact API, greatly improving performance.

Note that users of this feature should probably use just the base EVCard API on the provided EContact objects, to avoid extra processing and caching in the EContact code.

Note: We previously considered implementing e_book_get_book_view()'s requested fields parameter, but that changes the whole view, though we only really want to change what is sent for changed/new contacts.

Note: Chris discovered a dedicated e_book_get_contact_uids() API in the Fremantle EDS fork. The new API call was accepted upstream (in master) and [https://bugzilla.gnome.org/show_bug.cgi?id=651446#c11 backported as a patch] (used in the Meego package) for the gnome-2-32 branch, but this is not a full solution.

[https://bugzilla.gnome.org/show_bug.cgi?id=652172 Upstream evolution-data-server bug]

== Optional Parsing of ebook Data in Client ==

The EDS client-side APIs currently parse and (temporarily) store the parsed details of VCards. However, this is often unnecessary because APIs such as e_book_get_book_view() cause it to return the whole VCard anyway. There will be some performance gain from delaying VCard parsing until it is really needed by calls to functions such as e_contact_new_from_vcard().

libebook provides different functionality:
# exchange data as vCard with EDS daemon
# parse items and wrap them in a C API

QtContacts replaces the second part. An obvious performance improvement is to only do the second part if the additional C APIs are really called. The following calls then transfer data without any additional parsing in libebook:
* e_book_get_contact()/asynchronous read/view + e_vcard_to_string()
* e_contact_new_from_vcard() + e_book_add_contact()/e_book_commit_contact()/asynchronous storing

Note that libecal (for iCalendar) can already do this, though libebook can't yet.

TODO:
# add delayed parsing to libebook, triggered by API call which needs parsed data

PROBLEM:
# If delayed parsing fails, how can error be reported?
# Should delayed parsing be done always (e_vcard_to_string())) or only if requested? How would that be configured by a libebook user?

An [http://build.meego.com/package/view_file?file=EBookView-Implement-delayed-vcard-parsing.patch&package=evolution-data-server&project=eds&srcmd5=977d2b885bed818cb616599749407e79 e_book_view_set_parse_vcards() patch], already used in the Meego package, partly implements this already. This patch was taken from an old fremantle EDS patch that was never actually used, and is generally considered to be a bad hack that should not be accepted upstream.

[https://bugzilla.gnome.org/show_bug.cgi?id=652173 Upstream evolution-data-server bug]

== Asynchronous API for storing items in libecal ==

libecal should allow the caller to request storing of data without blocking on processing of the data. It should notify the caller when the data has been stored, or when processing has failed.

These functions may require async versions:
* icalcomponent_new_from_string()
* e_cal_modify_object[_with_mod]
* e_cal_create_object()

Making libical asynchronous is almost impossible without going multithreaded. Let's focus on e_cal D-Bus communication first.

Rob started some work on this in the rbradford-wip-ecal-async-api branch. Perhaps you can reuse some of that work. It is not functional, but might be a source of inspiration or code fragments.

[https://bugzilla.gnome.org/show_bug.cgi?id=652174 Upstream evolution-data-server bug]

== More Efficient Access to Meta Data for Change Tracking ==

This was discussed a while ago: [[http://www.mail-archive.com/evolution-hackers@gnome.org/msg03029.html
[Evolution-hackers] concurrent modifications of items in GUI and EDS database]]

Data synchronization only needs to know which items exist (list of local IDs) and which revision of each item is currently stored. Revisions can be identified with a string which changes for each modification. Restoring an older revision should (but doesn't have to) restore the revision string.

For calendar, UID+RECURRENCE-ID and LAST-MODIFIED can be used. For contacts, UID + REV.

By comparing a stored list against the current one, changes can be detected without relying on journals in the daemon or comparison of time stamps. Journals are hard to maintain (how many of them, how long?). Comparison of time stamps has problems when changes are allowed while a sync runs and assumes that system time is linear (which is not always true).

As an additional use case, local ID + revision string could be used to prevent overwriting stale data in the server, as in HTTP PUT eTag checks.

TODO:

The Maemo Fremantle port of EDS, which is of course already open source, apparently has some solution for this, according to Mathias Hasselman, so that should be investigated first.

However, this should probably be made possible by:

# Extending the EDS query language with a flag that limits the delivered data to just:
## UID and REV for libebook (REV is in VCard 3.0)
## UID+RECURRENCE-ID+LAST-MODIFIED, for libecal
# Implementing these queries for the local contact and calendar backends. The query would be rejected, with an error, for other backends.

[https://bugzilla.gnome.org/show_bug.cgi?id=652179 libebook: Upstream evolution-data-server bug] 
[https://bugzilla.gnome.org/show_bug.cgi?id=652180 libecal: Upstream evolution-data-server bug]

== Atomic Updates ==

Later, implement atomic changes as discussed in the mail thread: [http://www.mail-archive.com/evolution-hackers@gnome.org/msg03029.html]

== Quick check for "data changed" ==

The libebook and libical APIs should offer a way to discover if any data whatsoever has changed. This would, for instance, help SyncEvolution to avoid unnecessary detailed checking. This could just check the modification dates of local storage files, such as the BDB .db file, or .ics file, as long as we do not use those files to store permanent caches of parsed data. This would be much quicker than listing items for the whole database and comparing their revisions strings, as is currently necessary.

This would require small API additions, to check that the opaque revision strings are unchanged, such as

gchar* e_book_get_revision(EBook *book)
gboolean e_book_check_revision(EBook *book , const gchar* revision)

and

gchar* e_cal_get_revision(ECal *ecal)
gboolean e_cal_check_revision(ECal ecal, const gchar revision)

The semantic would be this:
* If the revision string is empty or different, then the content of the database '''might''' have changed. A more detailed check is necessary to determine if it has really changed. This will be the case for non-local storage.
* If it is non-empty and the same as before, the nothing has changed. No further check will be necessary.

[https://bugzilla.gnome.org/show_bug.cgi?id=652175 libebook: Upstream evolution-data-server bug] 
[https://bugzilla.gnome.org/show_bug.cgi?id=652177 libecal: Upstream evolution-data-server bug]

== Contacts: Store PHOTO Data as Plain Files ==

Photos should be transferred over D-Bus via a filepath to a local file rather than transferring the actual photo data over D-Bus. The VCard format already allows this option (See the PHOTO property). We must be careful to keep these local files in sync and to remove them when contacts are removed, and to restrict access to the files from other processes.

There should be utility code available to libebook users to convert between the different representations.

Goals:
# Apps should be able to create photo files without ever passing the data to libebook.
# External PHOTO file and corresponding contact must be kept in sync:
## If a contact is deleted, the photo also needs to be removed.
## If a photo file is created and then storing the contact fails, who removes the file?
# Access to photo data must be limited to processes which have the right permissions.

Proposal:
* For each ~/.local/share/evolution/foo calendar, store photos in <photo path>=~/.local/share/evolution/foo/photos, with same permissions.
* New photo files are created with a [http://en.wikipedia.org/wiki/Universally_unique_identifier UUID] as base name and suffix according to the file content.
* If a contact is deleted and has a PHOTO;VALUE=uri:file://<photo path>/*, delete that file.
* For apps which want to create the photo file directly:
** Apps get a file descriptor and the path from a new libebook API.
** They own the file until the EDS server acknowledges the successful storing of a contact with a matching PHOTO;VALUE=uri:file property. If that fails, the app must remove the file.
* Add new API calls to libebook which split out resp. include data inline.
** Must track ownership of file after splitting it out and before storing - might never get stored!

Open:
* Should there be automatic garbage collection of stale photos in case something went wrong and a photo wasn't properly deleted?
* Trust TYPE=JPEG/PNG/... or look into file content to determine type?

[https://bugzilla.gnome.org/show_bug.cgi?id=652178 Upstream evolution-data-server bug]

== Future: Content Protection ==

Only authorized applications (= processes) with the necessary permissions should be able to read/write data. Exact permissions and mechanisms for verifying them to be defined as part of the (revised?) MeeGo architecture.

This can be added as follows:
# lock down file permissions
# add permission checks to EDS D-Bus servers

Currently not planned. Depends on the future MeeGo security infrastructure.

Architecture/planning/evolution-data-server/eds-improvements

2011-06-09T10:22:12Z

Murrayc: /* More Efficient Access to Meta Data for Change Tracking */

This page describes improvements that are desirable or even essential for good performance of Evolution Data Server (EDS) as PIM storage in MeeGo.

All of these changes should be developed for the evolution-data-server gnome-2-32 branch, as used by MeeGo 1.2, then ported to master for review by upstream maintainers. Most changes will not be accepted in upstream's gnome-2-32 branch, because is is stable, but Meego will apply them as patches to its package.

== Change Tracking ==

EDS is based on the model/view/controller principle: an application opens a database (calendar or addressbook) and requests a view with server-side filtering, then the application is sent:
# all data covered by the view
# the "all data sent" signal
# changes (add/update/remove) while the view exists

Applications modify data with asynchronous requests (controller) and keep showing the old data until the change notification comes in. The exact changes made to the data are determined by the server (for example when meeting invitations are sent).

QtContacts and KCalCore do not have such a strict model. With QtContacts the app simply loads and stores contacts and is responsible for maintaining its own view. Changes are sent for changes made after the database was opened. KCalCore is based on the concept of loading a calendar (partially) into the client, modifying it there, then writing it back. Changes are also sent while the storage is open.

TODO:
* https://bugs.meego.com/show_bug.cgi?id=18760 - "QtContacts-EDS + EDS: more flexible EBookView (UID only, only future changes)"

=== Suppress Unnecessary Data Transfer ===

EDS currently sends all data when a view is requested, which is unnecessary when the client is only interested in change notifications. An extension to the EDS query language could optionally suppresses that sending of existing data when the view is opened, to be implemented only for the local contacts and calendar backends. This would allow easier integration with the QContact API.

The Fremantle EDS fork also had a “freezable” (static) capability for libebook, which should probably be investigated. See, for instance: http://maemo.org/api_refs/5.0/5.0-final/libebook/EBookView.html#e-book-view-set-freezable

One possibility for adding the flag is to (ab)use the existing query language with a match for special fields:
* in the client: query = e_book_query_orv(e_book_query_field_exists("X-EDS-QUERY-FLAG-SKIP-DATA-DUMP"), <real query>, NULL);
* in the server: "if (query is OR-Query and first clause is exists("X-EDS-QUERY-FLAG-SKIP-DATA-DUMP")) { skip_data_dump = TRUE; query = second clause; }

[https://bugzilla.gnome.org/show_bug.cgi?id=652171 Upstream evolution-data-server bug]

=== Nofify with UIDs Only ===

EBookView emits signals when contacts are changed, added, or removed. But those signals sends lists of entire EContacts, containing the entire contact data, though we often have all the data already and just want to know which contacts have changed.

[http://developer.gnome.org/libebook/stable/EBook.html#e-book-get-book-view e_book_get_book_view()] takes an [http://developer.gnome.org/libebook/stable/libebook-e-book-query.html#EBookQuery EBookQuery] which could be extended to take a new flag to make the EBookView send only the UID in the signals' EContacts. For instance, "X-EDS-QUERY-FLAG-NOTIFY-UID-ONLY". The UID is already stored separately for use as the primary key of the local-storage backend's BDB database, so no VCard parsing should be necessary to provide it.

This would allow easier integration with the QContact API, greatly improving performance.

Note that users of this feature should probably use just the base EVCard API on the provided EContact objects, to avoid extra processing and caching in the EContact code.

Note: We previously considered implementing e_book_get_book_view()'s requested fields parameter, but that changes the whole view, though we only really want to change what is sent for changed/new contacts.

Note: Chris discovered a dedicated e_book_get_contact_uids() API in the Fremantle EDS fork. The new API call was accepted upstream (in master) and [https://bugzilla.gnome.org/show_bug.cgi?id=651446#c11 backported as a patch] (used in the Meego package) for the gnome-2-32 branch, but this is not a full solution.

[https://bugzilla.gnome.org/show_bug.cgi?id=652172 Upstream evolution-data-server bug]

== Optional Parsing of ebook Data in Client ==

The EDS client-side APIs currently parse and (temporarily) store the parsed details of VCards. However, this is often unnecessary because APIs such as e_book_get_book_view() cause it to return the whole VCard anyway. There will be some performance gain from delaying VCard parsing until it is really needed by calls to functions such as e_contact_new_from_vcard().

libebook provides different functionality:
# exchange data as vCard with EDS daemon
# parse items and wrap them in a C API

QtContacts replaces the second part. An obvious performance improvement is to only do the second part if the additional C APIs are really called. The following calls then transfer data without any additional parsing in libebook:
* e_book_get_contact()/asynchronous read/view + e_vcard_to_string()
* e_contact_new_from_vcard() + e_book_add_contact()/e_book_commit_contact()/asynchronous storing

Note that libecal (for iCalendar) can already do this, though libebook can't yet.

TODO:
# add delayed parsing to libebook, triggered by API call which needs parsed data

PROBLEM:
# If delayed parsing fails, how can error be reported?
# Should delayed parsing be done always (e_vcard_to_string())) or only if requested? How would that be configured by a libebook user?

An [http://build.meego.com/package/view_file?file=EBookView-Implement-delayed-vcard-parsing.patch&package=evolution-data-server&project=eds&srcmd5=977d2b885bed818cb616599749407e79 e_book_view_set_parse_vcards() patch], already used in the Meego package, partly implements this already. This patch was taken from an old fremantle EDS patch that was never actually used, and is generally considered to be a bad hack that should not be accepted upstream.

[https://bugzilla.gnome.org/show_bug.cgi?id=652173 Upstream evolution-data-server bug]

== Asynchronous API for storing items in libecal ==

libecal should allow the caller to request storing of data without blocking on processing of the data. It should notify the caller when the data has been stored, or when processing has failed.

These functions may require async versions:
* icalcomponent_new_from_string()
* e_cal_modify_object[_with_mod]
* e_cal_create_object()

Making libical asynchronous is almost impossible without going multithreaded. Let's focus on e_cal D-Bus communication first.

Rob started some work on this in the rbradford-wip-ecal-async-api branch. Perhaps you can reuse some of that work. It is not functional, but might be a source of inspiration or code fragments.

[https://bugzilla.gnome.org/show_bug.cgi?id=652174 Upstream evolution-data-server bug]

== More Efficient Access to Meta Data for Change Tracking ==

This was discussed a while ago: [[http://www.mail-archive.com/evolution-hackers@gnome.org/msg03029.html
[Evolution-hackers] concurrent modifications of items in GUI and EDS database]]

Data synchronization only needs to know which items exist (list of local IDs) and which revision of each item is currently stored. Revisions can be identified with a string which changes for each modification. Restoring an older revision should (but doesn't have to) restore the revision string.

For calendar, UID+RECURRENCE-ID and LAST-MODIFIED can be used. For contacts, UID + REV.

By comparing a stored list against the current one, changes can be detected without relying on journals in the daemon or comparison of time stamps. Journals are hard to maintain (how many of them, how long?). Comparison of time stamps has problems when changes are allowed while a sync runs and assumes that system time is linear (which is not always true).

As an additional use case, local ID + revision string could be used to prevent overwriting stale data in the server, as in HTTP PUT eTag checks.

TODO:

The Maemo Fremantle port of EDS, which is of course already open source, apparently has some solution for this, according to Mathias Hasselman, so that should be investigated first.

However, this should probably be made possible by:

# Extending the EDS query language with a flag that limits the delivered data to just:
## UID and REV for libebook (REV is in VCard 3.0)
## UID+RECURRENCE-ID+LAST-MODIFIED, for libecal
# Implementing these queries for the local contact and calendar backends. The query would be rejected, with an error, for other backends.

[https://bugzilla.gnome.org/show_bug.cgi?id=652179 libebook: Upstream evolution-data-server bug] 
[https://bugzilla.gnome.org/show_bug.cgi?id=652180 libecal: Upstream evolution-data-server bug]

== Atomic Updates ==

Later, implement atomic changes as discussed in the mail thread: [http://www.mail-archive.com/evolution-hackers@gnome.org/msg03029.html]

== Quick check for "data changed" ==

The libebook and libical APIs should offer a way to discover if any data whatsoever has changed. This would, for instance, help SyncEvolution to avoid unnecessary detailed checking. This could just check the modification dates of local storage files, such as the BDB .db file, or .ics file, as long as we do not use those files to store permanent caches of parsed data. This would be much quicker than listing items for the whole database and comparing their revisions strings, as is currently necessary.

This would require small API additions, to check that the opaque revision strings are unchanged, such as

gchar* e_book_get_revision(EBook *book)
gboolean e_book_check_revision(EBook *book , const gchar* revision)

and

gchar* e_cal_get_revision(ECal *ecal)
gboolean e_cal_check_revision(ECal ecal, const gchar revision)

The semantic would be this:
* If the revision string is empty or different, then the content of the database '''might''' have changed. A more detailed check is necessary to determine if it has really changed. This will be the case for non-local storage.
* If it is non-empty and the same as before, the nothing has changed. No further check will be necessary.

[https://bugzilla.gnome.org/show_bug.cgi?id=652175 libebook: Upstream evolution-data-server bug] 
[https://bugzilla.gnome.org/show_bug.cgi?id=652177 libecal: Upstream evolution-data-server bug]

== Contacts: Store PHOTO Data as Plain Files ==

Photos should be transferred over D-Bus via a filepath to a local file rather than transferring the actual photo data over D-Bus. The VCard format already allows this option (See the PHOTO property). We must be careful to keep these local files in sync and to remove them when contacts are removed, and to restrict access to the files from other processes.

There should be utility code available to libebook users to convert between the different representations.

Goals:
# Apps should be able to create photo files without ever passing the data to libebook.
# External PHOTO file and corresponding contact must be kept in sync:
## If a contact is deleted, the photo also needs to be removed.
## If a photo file is created and then storing the contact fails, who removes the file?
# Access to photo data must be limited to processes which have the right permissions.

Proposal:
* For each ~/.local/share/evolution/foo calendar, store photos in <photo path>=~/.local/share/evolution/foo/photos, with same permissions.
* New photo files are created with a [http://en.wikipedia.org/wiki/Universally_unique_identifier UUID] as base name and suffix according to the file content.
* If a contact is deleted and has a PHOTO;VALUE=uri:file://<photo path>/*, delete that file.
* For apps which want to create the photo file directly:
** Apps get a file descriptor and the path from a new libebook API.
** They own the file until the EDS server acknowledges the successful storing of a contact with a matching PHOTO;VALUE=uri:file property. If that fails, the app must remove the file.
* Add new API calls to libebook which split out resp. include data inline.
** Must track ownership of file after splitting it out and before storing - might never get stored!

Open:
* Should there be automatic garbage collection of stale photos in case something went wrong and a photo wasn't properly deleted?
* Trust TYPE=JPEG/PNG/... or look into file content to determine type?

[https://bugzilla.gnome.org/show_bug.cgi?id=652178 Upstream evolution-data-server bug]

== Future: Content Protection ==

Only authorized applications (= processes) with the necessary permissions should be able to read/write data. Exact permissions and mechanisms for verifying them to be defined as part of the (revised?) MeeGo architecture.

This can be added as follows:
# lock down file permissions
# add permission checks to EDS D-Bus servers

Currently not planned. Depends on the future MeeGo security infrastructure.

Architecture/planning/evolution-data-server/eds-improvements

2011-06-09T10:21:59Z

Murrayc: /* More Efficient Access to Meta Data for Change Tracking */

This page describes improvements that are desirable or even essential for good performance of Evolution Data Server (EDS) as PIM storage in MeeGo.

All of these changes should be developed for the evolution-data-server gnome-2-32 branch, as used by MeeGo 1.2, then ported to master for review by upstream maintainers. Most changes will not be accepted in upstream's gnome-2-32 branch, because is is stable, but Meego will apply them as patches to its package.

== Change Tracking ==

EDS is based on the model/view/controller principle: an application opens a database (calendar or addressbook) and requests a view with server-side filtering, then the application is sent:
# all data covered by the view
# the "all data sent" signal
# changes (add/update/remove) while the view exists

Applications modify data with asynchronous requests (controller) and keep showing the old data until the change notification comes in. The exact changes made to the data are determined by the server (for example when meeting invitations are sent).

QtContacts and KCalCore do not have such a strict model. With QtContacts the app simply loads and stores contacts and is responsible for maintaining its own view. Changes are sent for changes made after the database was opened. KCalCore is based on the concept of loading a calendar (partially) into the client, modifying it there, then writing it back. Changes are also sent while the storage is open.

TODO:
* https://bugs.meego.com/show_bug.cgi?id=18760 - "QtContacts-EDS + EDS: more flexible EBookView (UID only, only future changes)"

=== Suppress Unnecessary Data Transfer ===

EDS currently sends all data when a view is requested, which is unnecessary when the client is only interested in change notifications. An extension to the EDS query language could optionally suppresses that sending of existing data when the view is opened, to be implemented only for the local contacts and calendar backends. This would allow easier integration with the QContact API.

The Fremantle EDS fork also had a “freezable” (static) capability for libebook, which should probably be investigated. See, for instance: http://maemo.org/api_refs/5.0/5.0-final/libebook/EBookView.html#e-book-view-set-freezable

One possibility for adding the flag is to (ab)use the existing query language with a match for special fields:
* in the client: query = e_book_query_orv(e_book_query_field_exists("X-EDS-QUERY-FLAG-SKIP-DATA-DUMP"), <real query>, NULL);
* in the server: "if (query is OR-Query and first clause is exists("X-EDS-QUERY-FLAG-SKIP-DATA-DUMP")) { skip_data_dump = TRUE; query = second clause; }

[https://bugzilla.gnome.org/show_bug.cgi?id=652171 Upstream evolution-data-server bug]

=== Nofify with UIDs Only ===

EBookView emits signals when contacts are changed, added, or removed. But those signals sends lists of entire EContacts, containing the entire contact data, though we often have all the data already and just want to know which contacts have changed.

[http://developer.gnome.org/libebook/stable/EBook.html#e-book-get-book-view e_book_get_book_view()] takes an [http://developer.gnome.org/libebook/stable/libebook-e-book-query.html#EBookQuery EBookQuery] which could be extended to take a new flag to make the EBookView send only the UID in the signals' EContacts. For instance, "X-EDS-QUERY-FLAG-NOTIFY-UID-ONLY". The UID is already stored separately for use as the primary key of the local-storage backend's BDB database, so no VCard parsing should be necessary to provide it.

This would allow easier integration with the QContact API, greatly improving performance.

Note that users of this feature should probably use just the base EVCard API on the provided EContact objects, to avoid extra processing and caching in the EContact code.

Note: We previously considered implementing e_book_get_book_view()'s requested fields parameter, but that changes the whole view, though we only really want to change what is sent for changed/new contacts.

Note: Chris discovered a dedicated e_book_get_contact_uids() API in the Fremantle EDS fork. The new API call was accepted upstream (in master) and [https://bugzilla.gnome.org/show_bug.cgi?id=651446#c11 backported as a patch] (used in the Meego package) for the gnome-2-32 branch, but this is not a full solution.

[https://bugzilla.gnome.org/show_bug.cgi?id=652172 Upstream evolution-data-server bug]

== Optional Parsing of ebook Data in Client ==

The EDS client-side APIs currently parse and (temporarily) store the parsed details of VCards. However, this is often unnecessary because APIs such as e_book_get_book_view() cause it to return the whole VCard anyway. There will be some performance gain from delaying VCard parsing until it is really needed by calls to functions such as e_contact_new_from_vcard().

libebook provides different functionality:
# exchange data as vCard with EDS daemon
# parse items and wrap them in a C API

QtContacts replaces the second part. An obvious performance improvement is to only do the second part if the additional C APIs are really called. The following calls then transfer data without any additional parsing in libebook:
* e_book_get_contact()/asynchronous read/view + e_vcard_to_string()
* e_contact_new_from_vcard() + e_book_add_contact()/e_book_commit_contact()/asynchronous storing

Note that libecal (for iCalendar) can already do this, though libebook can't yet.

TODO:
# add delayed parsing to libebook, triggered by API call which needs parsed data

PROBLEM:
# If delayed parsing fails, how can error be reported?
# Should delayed parsing be done always (e_vcard_to_string())) or only if requested? How would that be configured by a libebook user?

An [http://build.meego.com/package/view_file?file=EBookView-Implement-delayed-vcard-parsing.patch&package=evolution-data-server&project=eds&srcmd5=977d2b885bed818cb616599749407e79 e_book_view_set_parse_vcards() patch], already used in the Meego package, partly implements this already. This patch was taken from an old fremantle EDS patch that was never actually used, and is generally considered to be a bad hack that should not be accepted upstream.

[https://bugzilla.gnome.org/show_bug.cgi?id=652173 Upstream evolution-data-server bug]

== Asynchronous API for storing items in libecal ==

libecal should allow the caller to request storing of data without blocking on processing of the data. It should notify the caller when the data has been stored, or when processing has failed.

These functions may require async versions:
* icalcomponent_new_from_string()
* e_cal_modify_object[_with_mod]
* e_cal_create_object()

Making libical asynchronous is almost impossible without going multithreaded. Let's focus on e_cal D-Bus communication first.

Rob started some work on this in the rbradford-wip-ecal-async-api branch. Perhaps you can reuse some of that work. It is not functional, but might be a source of inspiration or code fragments.

[https://bugzilla.gnome.org/show_bug.cgi?id=652174 Upstream evolution-data-server bug]

== More Efficient Access to Meta Data for Change Tracking ==

This was discussed a while ago: [[http://www.mail-archive.com/evolution-hackers@gnome.org/msg03029.html
[Evolution-hackers] concurrent modifications of items in GUI and EDS database]]

Data synchronization only needs to know which items exist (list of local IDs) and which revision of each item is currently stored. Revisions can be identified with a string which changes for each modification. Restoring an older revision should (but doesn't have to) restore the revision string.

For calendar, UID+RECURRENCE-ID and LAST-MODIFIED can be used. For contacts, UID + REV.

By comparing a stored list against the current one, changes can be detected without relying on journals in the daemon or comparison of time stamps. Journals are hard to maintain (how many of them, how long?). Comparison of time stamps has problems when changes are allowed while a sync runs and assumes that system time is linear (which is not always true).

As an additional use case, local ID + revision string could be used to prevent overwriting stale data in the server, as in HTTP PUT eTag checks.

TODO:

The Maemo Fremantle port of EDS, which is of course already open source, apparently has some solution for this, according to Mathias Hasselman, so that should be investigated first.

However, this should probably be made possible by:

# Extending the EDS query language with a flag that limits the delivered data to just:
## UID and REV for libebook (REV is in VCard 3.0)
## UID+RECURRENCE-ID+LAST-MODIFIED, for libecal
# Implementing these queries for the local contact and calendar backends. The query would be rejected, with an error, for other backends.

[https://bugzilla.gnome.org/show_bug.cgi?id=652179 libebook: Upstream evolution-data-server bug]
[https://bugzilla.gnome.org/show_bug.cgi?id=652180 libecal: Upstream evolution-data-server bug]

== Atomic Updates ==

Later, implement atomic changes as discussed in the mail thread: [http://www.mail-archive.com/evolution-hackers@gnome.org/msg03029.html]

== Quick check for "data changed" ==

The libebook and libical APIs should offer a way to discover if any data whatsoever has changed. This would, for instance, help SyncEvolution to avoid unnecessary detailed checking. This could just check the modification dates of local storage files, such as the BDB .db file, or .ics file, as long as we do not use those files to store permanent caches of parsed data. This would be much quicker than listing items for the whole database and comparing their revisions strings, as is currently necessary.

This would require small API additions, to check that the opaque revision strings are unchanged, such as

gchar* e_book_get_revision(EBook *book)
gboolean e_book_check_revision(EBook *book , const gchar* revision)

and

gchar* e_cal_get_revision(ECal *ecal)
gboolean e_cal_check_revision(ECal ecal, const gchar revision)

The semantic would be this:
* If the revision string is empty or different, then the content of the database '''might''' have changed. A more detailed check is necessary to determine if it has really changed. This will be the case for non-local storage.
* If it is non-empty and the same as before, the nothing has changed. No further check will be necessary.

[https://bugzilla.gnome.org/show_bug.cgi?id=652175 libebook: Upstream evolution-data-server bug] 
[https://bugzilla.gnome.org/show_bug.cgi?id=652177 libecal: Upstream evolution-data-server bug]

== Contacts: Store PHOTO Data as Plain Files ==

Photos should be transferred over D-Bus via a filepath to a local file rather than transferring the actual photo data over D-Bus. The VCard format already allows this option (See the PHOTO property). We must be careful to keep these local files in sync and to remove them when contacts are removed, and to restrict access to the files from other processes.

There should be utility code available to libebook users to convert between the different representations.

Goals:
# Apps should be able to create photo files without ever passing the data to libebook.
# External PHOTO file and corresponding contact must be kept in sync:
## If a contact is deleted, the photo also needs to be removed.
## If a photo file is created and then storing the contact fails, who removes the file?
# Access to photo data must be limited to processes which have the right permissions.

Proposal:
* For each ~/.local/share/evolution/foo calendar, store photos in <photo path>=~/.local/share/evolution/foo/photos, with same permissions.
* New photo files are created with a [http://en.wikipedia.org/wiki/Universally_unique_identifier UUID] as base name and suffix according to the file content.
* If a contact is deleted and has a PHOTO;VALUE=uri:file://<photo path>/*, delete that file.
* For apps which want to create the photo file directly:
** Apps get a file descriptor and the path from a new libebook API.
** They own the file until the EDS server acknowledges the successful storing of a contact with a matching PHOTO;VALUE=uri:file property. If that fails, the app must remove the file.
* Add new API calls to libebook which split out resp. include data inline.
** Must track ownership of file after splitting it out and before storing - might never get stored!

Open:
* Should there be automatic garbage collection of stale photos in case something went wrong and a photo wasn't properly deleted?
* Trust TYPE=JPEG/PNG/... or look into file content to determine type?

[https://bugzilla.gnome.org/show_bug.cgi?id=652178 Upstream evolution-data-server bug]

== Future: Content Protection ==

Only authorized applications (= processes) with the necessary permissions should be able to read/write data. Exact permissions and mechanisms for verifying them to be defined as part of the (revised?) MeeGo architecture.

This can be added as follows:
# lock down file permissions
# add permission checks to EDS D-Bus servers

Currently not planned. Depends on the future MeeGo security infrastructure.

Architecture/planning/evolution-data-server/eds-improvements

2011-06-09T10:13:06Z

Murrayc: /* Atomic Updates */

This page describes improvements that are desirable or even essential for good performance of Evolution Data Server (EDS) as PIM storage in MeeGo.

All of these changes should be developed for the evolution-data-server gnome-2-32 branch, as used by MeeGo 1.2, then ported to master for review by upstream maintainers. Most changes will not be accepted in upstream's gnome-2-32 branch, because is is stable, but Meego will apply them as patches to its package.

== Change Tracking ==

EDS is based on the model/view/controller principle: an application opens a database (calendar or addressbook) and requests a view with server-side filtering, then the application is sent:
# all data covered by the view
# the "all data sent" signal
# changes (add/update/remove) while the view exists

Applications modify data with asynchronous requests (controller) and keep showing the old data until the change notification comes in. The exact changes made to the data are determined by the server (for example when meeting invitations are sent).

QtContacts and KCalCore do not have such a strict model. With QtContacts the app simply loads and stores contacts and is responsible for maintaining its own view. Changes are sent for changes made after the database was opened. KCalCore is based on the concept of loading a calendar (partially) into the client, modifying it there, then writing it back. Changes are also sent while the storage is open.

TODO:
* https://bugs.meego.com/show_bug.cgi?id=18760 - "QtContacts-EDS + EDS: more flexible EBookView (UID only, only future changes)"

=== Suppress Unnecessary Data Transfer ===

EDS currently sends all data when a view is requested, which is unnecessary when the client is only interested in change notifications. An extension to the EDS query language could optionally suppresses that sending of existing data when the view is opened, to be implemented only for the local contacts and calendar backends. This would allow easier integration with the QContact API.

The Fremantle EDS fork also had a “freezable” (static) capability for libebook, which should probably be investigated. See, for instance: http://maemo.org/api_refs/5.0/5.0-final/libebook/EBookView.html#e-book-view-set-freezable

One possibility for adding the flag is to (ab)use the existing query language with a match for special fields:
* in the client: query = e_book_query_orv(e_book_query_field_exists("X-EDS-QUERY-FLAG-SKIP-DATA-DUMP"), <real query>, NULL);
* in the server: "if (query is OR-Query and first clause is exists("X-EDS-QUERY-FLAG-SKIP-DATA-DUMP")) { skip_data_dump = TRUE; query = second clause; }

[https://bugzilla.gnome.org/show_bug.cgi?id=652171 Upstream evolution-data-server bug]

=== Nofify with UIDs Only ===

EBookView emits signals when contacts are changed, added, or removed. But those signals sends lists of entire EContacts, containing the entire contact data, though we often have all the data already and just want to know which contacts have changed.

[http://developer.gnome.org/libebook/stable/EBook.html#e-book-get-book-view e_book_get_book_view()] takes an [http://developer.gnome.org/libebook/stable/libebook-e-book-query.html#EBookQuery EBookQuery] which could be extended to take a new flag to make the EBookView send only the UID in the signals' EContacts. For instance, "X-EDS-QUERY-FLAG-NOTIFY-UID-ONLY". The UID is already stored separately for use as the primary key of the local-storage backend's BDB database, so no VCard parsing should be necessary to provide it.

This would allow easier integration with the QContact API, greatly improving performance.

Note that users of this feature should probably use just the base EVCard API on the provided EContact objects, to avoid extra processing and caching in the EContact code.

Note: We previously considered implementing e_book_get_book_view()'s requested fields parameter, but that changes the whole view, though we only really want to change what is sent for changed/new contacts.

Note: Chris discovered a dedicated e_book_get_contact_uids() API in the Fremantle EDS fork. The new API call was accepted upstream (in master) and [https://bugzilla.gnome.org/show_bug.cgi?id=651446#c11 backported as a patch] (used in the Meego package) for the gnome-2-32 branch, but this is not a full solution.

[https://bugzilla.gnome.org/show_bug.cgi?id=652172 Upstream evolution-data-server bug]

== Optional Parsing of ebook Data in Client ==

The EDS client-side APIs currently parse and (temporarily) store the parsed details of VCards. However, this is often unnecessary because APIs such as e_book_get_book_view() cause it to return the whole VCard anyway. There will be some performance gain from delaying VCard parsing until it is really needed by calls to functions such as e_contact_new_from_vcard().

libebook provides different functionality:
# exchange data as vCard with EDS daemon
# parse items and wrap them in a C API

QtContacts replaces the second part. An obvious performance improvement is to only do the second part if the additional C APIs are really called. The following calls then transfer data without any additional parsing in libebook:
* e_book_get_contact()/asynchronous read/view + e_vcard_to_string()
* e_contact_new_from_vcard() + e_book_add_contact()/e_book_commit_contact()/asynchronous storing

Note that libecal (for iCalendar) can already do this, though libebook can't yet.

TODO:
# add delayed parsing to libebook, triggered by API call which needs parsed data

PROBLEM:
# If delayed parsing fails, how can error be reported?
# Should delayed parsing be done always (e_vcard_to_string())) or only if requested? How would that be configured by a libebook user?

An [http://build.meego.com/package/view_file?file=EBookView-Implement-delayed-vcard-parsing.patch&package=evolution-data-server&project=eds&srcmd5=977d2b885bed818cb616599749407e79 e_book_view_set_parse_vcards() patch], already used in the Meego package, partly implements this already. This patch was taken from an old fremantle EDS patch that was never actually used, and is generally considered to be a bad hack that should not be accepted upstream.

[https://bugzilla.gnome.org/show_bug.cgi?id=652173 Upstream evolution-data-server bug]

== Asynchronous API for storing items in libecal ==

libecal should allow the caller to request storing of data without blocking on processing of the data. It should notify the caller when the data has been stored, or when processing has failed.

These functions may require async versions:
* icalcomponent_new_from_string()
* e_cal_modify_object[_with_mod]
* e_cal_create_object()

Making libical asynchronous is almost impossible without going multithreaded. Let's focus on e_cal D-Bus communication first.

Rob started some work on this in the rbradford-wip-ecal-async-api branch. Perhaps you can reuse some of that work. It is not functional, but might be a source of inspiration or code fragments.

[https://bugzilla.gnome.org/show_bug.cgi?id=652174 Upstream evolution-data-server bug]

== More Efficient Access to Meta Data for Change Tracking ==

This was discussed a while ago: [[http://www.mail-archive.com/evolution-hackers@gnome.org/msg03029.html
[Evolution-hackers] concurrent modifications of items in GUI and EDS database]]

Data synchronization only needs to know which items exist (list of local IDs) and which revision of each item is currently stored. Revisions can be identified with a string which changes for each modification. Restoring an older revision should (but doesn't have to) restore the revision string.

For calendar, UID+RECURRENCE-ID and LAST-MODIFIED can be used. For contacts, UID + REV.

By comparing a stored list against the current one, changes can be detected without relying on journals in the daemon or comparison of time stamps. Journals are hard to maintain (how many of them, how long?). Comparison of time stamps has problems when changes are allowed while a sync runs and assumes that system time is linear (which is not always true).

As an additional use case, local ID + revision string could be used to prevent overwriting stale data in the server, as in HTTP PUT eTag checks.

TODO:

The Maemo Fremantle port of EDS, which is of course already open source, apparently has some solution for this, according to Mathias Hasselman, so that should be investigated first.

However, this should probably be made possible by:

# Extending the EDS query language with a flag that limits the delivered data to just:
## UID and REV for libebook (REV is in VCard 3.0)
## UID+RECURRENCE-ID+LAST-MODIFIED, for libecal
# Implementing these queries for the local contact and calendar backends. The query would be rejected, with an error, for other backends.

== Atomic Updates ==

Later, implement atomic changes as discussed in the mail thread: [http://www.mail-archive.com/evolution-hackers@gnome.org/msg03029.html]

== Quick check for "data changed" ==

The libebook and libical APIs should offer a way to discover if any data whatsoever has changed. This would, for instance, help SyncEvolution to avoid unnecessary detailed checking. This could just check the modification dates of local storage files, such as the BDB .db file, or .ics file, as long as we do not use those files to store permanent caches of parsed data. This would be much quicker than listing items for the whole database and comparing their revisions strings, as is currently necessary.

This would require small API additions, to check that the opaque revision strings are unchanged, such as

gchar* e_book_get_revision(EBook *book)
gboolean e_book_check_revision(EBook *book , const gchar* revision)

and

gchar* e_cal_get_revision(ECal *ecal)
gboolean e_cal_check_revision(ECal ecal, const gchar revision)

The semantic would be this:
* If the revision string is empty or different, then the content of the database '''might''' have changed. A more detailed check is necessary to determine if it has really changed. This will be the case for non-local storage.
* If it is non-empty and the same as before, the nothing has changed. No further check will be necessary.

[https://bugzilla.gnome.org/show_bug.cgi?id=652175 libebook: Upstream evolution-data-server bug] 
[https://bugzilla.gnome.org/show_bug.cgi?id=652177 libecal: Upstream evolution-data-server bug]

== Contacts: Store PHOTO Data as Plain Files ==

Photos should be transferred over D-Bus via a filepath to a local file rather than transferring the actual photo data over D-Bus. The VCard format already allows this option (See the PHOTO property). We must be careful to keep these local files in sync and to remove them when contacts are removed, and to restrict access to the files from other processes.

There should be utility code available to libebook users to convert between the different representations.

Goals:
# Apps should be able to create photo files without ever passing the data to libebook.
# External PHOTO file and corresponding contact must be kept in sync:
## If a contact is deleted, the photo also needs to be removed.
## If a photo file is created and then storing the contact fails, who removes the file?
# Access to photo data must be limited to processes which have the right permissions.

Proposal:
* For each ~/.local/share/evolution/foo calendar, store photos in <photo path>=~/.local/share/evolution/foo/photos, with same permissions.
* New photo files are created with a [http://en.wikipedia.org/wiki/Universally_unique_identifier UUID] as base name and suffix according to the file content.
* If a contact is deleted and has a PHOTO;VALUE=uri:file://<photo path>/*, delete that file.
* For apps which want to create the photo file directly:
** Apps get a file descriptor and the path from a new libebook API.
** They own the file until the EDS server acknowledges the successful storing of a contact with a matching PHOTO;VALUE=uri:file property. If that fails, the app must remove the file.
* Add new API calls to libebook which split out resp. include data inline.
** Must track ownership of file after splitting it out and before storing - might never get stored!

Open:
* Should there be automatic garbage collection of stale photos in case something went wrong and a photo wasn't properly deleted?
* Trust TYPE=JPEG/PNG/... or look into file content to determine type?

[https://bugzilla.gnome.org/show_bug.cgi?id=652178 Upstream evolution-data-server bug]

== Future: Content Protection ==

Only authorized applications (= processes) with the necessary permissions should be able to read/write data. Exact permissions and mechanisms for verifying them to be defined as part of the (revised?) MeeGo architecture.

This can be added as follows:
# lock down file permissions
# add permission checks to EDS D-Bus servers

Currently not planned. Depends on the future MeeGo security infrastructure.

Architecture/planning/evolution-data-server/eds-improvements

2011-06-09T10:12:41Z

Murrayc: /* Contacts: Store PHOTO Data as Plain Files */

This page describes improvements that are desirable or even essential for good performance of Evolution Data Server (EDS) as PIM storage in MeeGo.

All of these changes should be developed for the evolution-data-server gnome-2-32 branch, as used by MeeGo 1.2, then ported to master for review by upstream maintainers. Most changes will not be accepted in upstream's gnome-2-32 branch, because is is stable, but Meego will apply them as patches to its package.

== Change Tracking ==

EDS is based on the model/view/controller principle: an application opens a database (calendar or addressbook) and requests a view with server-side filtering, then the application is sent:
# all data covered by the view
# the "all data sent" signal
# changes (add/update/remove) while the view exists

Applications modify data with asynchronous requests (controller) and keep showing the old data until the change notification comes in. The exact changes made to the data are determined by the server (for example when meeting invitations are sent).

QtContacts and KCalCore do not have such a strict model. With QtContacts the app simply loads and stores contacts and is responsible for maintaining its own view. Changes are sent for changes made after the database was opened. KCalCore is based on the concept of loading a calendar (partially) into the client, modifying it there, then writing it back. Changes are also sent while the storage is open.

TODO:
* https://bugs.meego.com/show_bug.cgi?id=18760 - "QtContacts-EDS + EDS: more flexible EBookView (UID only, only future changes)"

=== Suppress Unnecessary Data Transfer ===

EDS currently sends all data when a view is requested, which is unnecessary when the client is only interested in change notifications. An extension to the EDS query language could optionally suppresses that sending of existing data when the view is opened, to be implemented only for the local contacts and calendar backends. This would allow easier integration with the QContact API.

The Fremantle EDS fork also had a “freezable” (static) capability for libebook, which should probably be investigated. See, for instance: http://maemo.org/api_refs/5.0/5.0-final/libebook/EBookView.html#e-book-view-set-freezable

One possibility for adding the flag is to (ab)use the existing query language with a match for special fields:
* in the client: query = e_book_query_orv(e_book_query_field_exists("X-EDS-QUERY-FLAG-SKIP-DATA-DUMP"), <real query>, NULL);
* in the server: "if (query is OR-Query and first clause is exists("X-EDS-QUERY-FLAG-SKIP-DATA-DUMP")) { skip_data_dump = TRUE; query = second clause; }

[https://bugzilla.gnome.org/show_bug.cgi?id=652171 Upstream evolution-data-server bug]

=== Nofify with UIDs Only ===

EBookView emits signals when contacts are changed, added, or removed. But those signals sends lists of entire EContacts, containing the entire contact data, though we often have all the data already and just want to know which contacts have changed.

[http://developer.gnome.org/libebook/stable/EBook.html#e-book-get-book-view e_book_get_book_view()] takes an [http://developer.gnome.org/libebook/stable/libebook-e-book-query.html#EBookQuery EBookQuery] which could be extended to take a new flag to make the EBookView send only the UID in the signals' EContacts. For instance, "X-EDS-QUERY-FLAG-NOTIFY-UID-ONLY". The UID is already stored separately for use as the primary key of the local-storage backend's BDB database, so no VCard parsing should be necessary to provide it.

This would allow easier integration with the QContact API, greatly improving performance.

Note that users of this feature should probably use just the base EVCard API on the provided EContact objects, to avoid extra processing and caching in the EContact code.

Note: We previously considered implementing e_book_get_book_view()'s requested fields parameter, but that changes the whole view, though we only really want to change what is sent for changed/new contacts.

Note: Chris discovered a dedicated e_book_get_contact_uids() API in the Fremantle EDS fork. The new API call was accepted upstream (in master) and [https://bugzilla.gnome.org/show_bug.cgi?id=651446#c11 backported as a patch] (used in the Meego package) for the gnome-2-32 branch, but this is not a full solution.

[https://bugzilla.gnome.org/show_bug.cgi?id=652172 Upstream evolution-data-server bug]

== Optional Parsing of ebook Data in Client ==

The EDS client-side APIs currently parse and (temporarily) store the parsed details of VCards. However, this is often unnecessary because APIs such as e_book_get_book_view() cause it to return the whole VCard anyway. There will be some performance gain from delaying VCard parsing until it is really needed by calls to functions such as e_contact_new_from_vcard().

libebook provides different functionality:
# exchange data as vCard with EDS daemon
# parse items and wrap them in a C API

QtContacts replaces the second part. An obvious performance improvement is to only do the second part if the additional C APIs are really called. The following calls then transfer data without any additional parsing in libebook:
* e_book_get_contact()/asynchronous read/view + e_vcard_to_string()
* e_contact_new_from_vcard() + e_book_add_contact()/e_book_commit_contact()/asynchronous storing

Note that libecal (for iCalendar) can already do this, though libebook can't yet.

TODO:
# add delayed parsing to libebook, triggered by API call which needs parsed data

PROBLEM:
# If delayed parsing fails, how can error be reported?
# Should delayed parsing be done always (e_vcard_to_string())) or only if requested? How would that be configured by a libebook user?

An [http://build.meego.com/package/view_file?file=EBookView-Implement-delayed-vcard-parsing.patch&package=evolution-data-server&project=eds&srcmd5=977d2b885bed818cb616599749407e79 e_book_view_set_parse_vcards() patch], already used in the Meego package, partly implements this already. This patch was taken from an old fremantle EDS patch that was never actually used, and is generally considered to be a bad hack that should not be accepted upstream.

[https://bugzilla.gnome.org/show_bug.cgi?id=652173 Upstream evolution-data-server bug]

== Asynchronous API for storing items in libecal ==

libecal should allow the caller to request storing of data without blocking on processing of the data. It should notify the caller when the data has been stored, or when processing has failed.

These functions may require async versions:
* icalcomponent_new_from_string()
* e_cal_modify_object[_with_mod]
* e_cal_create_object()

Making libical asynchronous is almost impossible without going multithreaded. Let's focus on e_cal D-Bus communication first.

Rob started some work on this in the rbradford-wip-ecal-async-api branch. Perhaps you can reuse some of that work. It is not functional, but might be a source of inspiration or code fragments.

[https://bugzilla.gnome.org/show_bug.cgi?id=652174 Upstream evolution-data-server bug]

== More Efficient Access to Meta Data for Change Tracking ==

This was discussed a while ago: [[http://www.mail-archive.com/evolution-hackers@gnome.org/msg03029.html
[Evolution-hackers] concurrent modifications of items in GUI and EDS database]]

Data synchronization only needs to know which items exist (list of local IDs) and which revision of each item is currently stored. Revisions can be identified with a string which changes for each modification. Restoring an older revision should (but doesn't have to) restore the revision string.

For calendar, UID+RECURRENCE-ID and LAST-MODIFIED can be used. For contacts, UID + REV.

By comparing a stored list against the current one, changes can be detected without relying on journals in the daemon or comparison of time stamps. Journals are hard to maintain (how many of them, how long?). Comparison of time stamps has problems when changes are allowed while a sync runs and assumes that system time is linear (which is not always true).

As an additional use case, local ID + revision string could be used to prevent overwriting stale data in the server, as in HTTP PUT eTag checks.

TODO:

The Maemo Fremantle port of EDS, which is of course already open source, apparently has some solution for this, according to Mathias Hasselman, so that should be investigated first.

However, this should probably be made possible by:

# Extending the EDS query language with a flag that limits the delivered data to just:
## UID and REV for libebook (REV is in VCard 3.0)
## UID+RECURRENCE-ID+LAST-MODIFIED, for libecal
# Implementing these queries for the local contact and calendar backends. The query would be rejected, with an error, for other backends.

== Atomic Updates ==

Later, implement atomic changes as discussed in the mail thread: [[http://www.mail-archive.com/evolution-hackers@gnome.org/msg03029.html

== Quick check for "data changed" ==

The libebook and libical APIs should offer a way to discover if any data whatsoever has changed. This would, for instance, help SyncEvolution to avoid unnecessary detailed checking. This could just check the modification dates of local storage files, such as the BDB .db file, or .ics file, as long as we do not use those files to store permanent caches of parsed data. This would be much quicker than listing items for the whole database and comparing their revisions strings, as is currently necessary.

This would require small API additions, to check that the opaque revision strings are unchanged, such as

gchar* e_book_get_revision(EBook *book)
gboolean e_book_check_revision(EBook *book , const gchar* revision)

and

gchar* e_cal_get_revision(ECal *ecal)
gboolean e_cal_check_revision(ECal ecal, const gchar revision)

The semantic would be this:
* If the revision string is empty or different, then the content of the database '''might''' have changed. A more detailed check is necessary to determine if it has really changed. This will be the case for non-local storage.
* If it is non-empty and the same as before, the nothing has changed. No further check will be necessary.

[https://bugzilla.gnome.org/show_bug.cgi?id=652175 libebook: Upstream evolution-data-server bug] 
[https://bugzilla.gnome.org/show_bug.cgi?id=652177 libecal: Upstream evolution-data-server bug]

== Contacts: Store PHOTO Data as Plain Files ==

Photos should be transferred over D-Bus via a filepath to a local file rather than transferring the actual photo data over D-Bus. The VCard format already allows this option (See the PHOTO property). We must be careful to keep these local files in sync and to remove them when contacts are removed, and to restrict access to the files from other processes.

There should be utility code available to libebook users to convert between the different representations.

Goals:
# Apps should be able to create photo files without ever passing the data to libebook.
# External PHOTO file and corresponding contact must be kept in sync:
## If a contact is deleted, the photo also needs to be removed.
## If a photo file is created and then storing the contact fails, who removes the file?
# Access to photo data must be limited to processes which have the right permissions.

Proposal:
* For each ~/.local/share/evolution/foo calendar, store photos in <photo path>=~/.local/share/evolution/foo/photos, with same permissions.
* New photo files are created with a [http://en.wikipedia.org/wiki/Universally_unique_identifier UUID] as base name and suffix according to the file content.
* If a contact is deleted and has a PHOTO;VALUE=uri:file://<photo path>/*, delete that file.
* For apps which want to create the photo file directly:
** Apps get a file descriptor and the path from a new libebook API.
** They own the file until the EDS server acknowledges the successful storing of a contact with a matching PHOTO;VALUE=uri:file property. If that fails, the app must remove the file.
* Add new API calls to libebook which split out resp. include data inline.
** Must track ownership of file after splitting it out and before storing - might never get stored!

Open:
* Should there be automatic garbage collection of stale photos in case something went wrong and a photo wasn't properly deleted?
* Trust TYPE=JPEG/PNG/... or look into file content to determine type?

[https://bugzilla.gnome.org/show_bug.cgi?id=652178 Upstream evolution-data-server bug]

== Future: Content Protection ==

Only authorized applications (= processes) with the necessary permissions should be able to read/write data. Exact permissions and mechanisms for verifying them to be defined as part of the (revised?) MeeGo architecture.

This can be added as follows:
# lock down file permissions
# add permission checks to EDS D-Bus servers

Currently not planned. Depends on the future MeeGo security infrastructure.

Architecture/planning/evolution-data-server/eds-improvements

2011-06-09T10:09:15Z

Murrayc: /* Contacts: Store PHOTO Data as Plain Files */

This page describes improvements that are desirable or even essential for good performance of Evolution Data Server (EDS) as PIM storage in MeeGo.

All of these changes should be developed for the evolution-data-server gnome-2-32 branch, as used by MeeGo 1.2, then ported to master for review by upstream maintainers. Most changes will not be accepted in upstream's gnome-2-32 branch, because is is stable, but Meego will apply them as patches to its package.

== Change Tracking ==

EDS is based on the model/view/controller principle: an application opens a database (calendar or addressbook) and requests a view with server-side filtering, then the application is sent:
# all data covered by the view
# the "all data sent" signal
# changes (add/update/remove) while the view exists

Applications modify data with asynchronous requests (controller) and keep showing the old data until the change notification comes in. The exact changes made to the data are determined by the server (for example when meeting invitations are sent).

QtContacts and KCalCore do not have such a strict model. With QtContacts the app simply loads and stores contacts and is responsible for maintaining its own view. Changes are sent for changes made after the database was opened. KCalCore is based on the concept of loading a calendar (partially) into the client, modifying it there, then writing it back. Changes are also sent while the storage is open.

TODO:
* https://bugs.meego.com/show_bug.cgi?id=18760 - "QtContacts-EDS + EDS: more flexible EBookView (UID only, only future changes)"

=== Suppress Unnecessary Data Transfer ===

EDS currently sends all data when a view is requested, which is unnecessary when the client is only interested in change notifications. An extension to the EDS query language could optionally suppresses that sending of existing data when the view is opened, to be implemented only for the local contacts and calendar backends. This would allow easier integration with the QContact API.

The Fremantle EDS fork also had a “freezable” (static) capability for libebook, which should probably be investigated. See, for instance: http://maemo.org/api_refs/5.0/5.0-final/libebook/EBookView.html#e-book-view-set-freezable

One possibility for adding the flag is to (ab)use the existing query language with a match for special fields:
* in the client: query = e_book_query_orv(e_book_query_field_exists("X-EDS-QUERY-FLAG-SKIP-DATA-DUMP"), <real query>, NULL);
* in the server: "if (query is OR-Query and first clause is exists("X-EDS-QUERY-FLAG-SKIP-DATA-DUMP")) { skip_data_dump = TRUE; query = second clause; }

[https://bugzilla.gnome.org/show_bug.cgi?id=652171 Upstream evolution-data-server bug]

=== Nofify with UIDs Only ===

EBookView emits signals when contacts are changed, added, or removed. But those signals sends lists of entire EContacts, containing the entire contact data, though we often have all the data already and just want to know which contacts have changed.

[http://developer.gnome.org/libebook/stable/EBook.html#e-book-get-book-view e_book_get_book_view()] takes an [http://developer.gnome.org/libebook/stable/libebook-e-book-query.html#EBookQuery EBookQuery] which could be extended to take a new flag to make the EBookView send only the UID in the signals' EContacts. For instance, "X-EDS-QUERY-FLAG-NOTIFY-UID-ONLY". The UID is already stored separately for use as the primary key of the local-storage backend's BDB database, so no VCard parsing should be necessary to provide it.

This would allow easier integration with the QContact API, greatly improving performance.

Note that users of this feature should probably use just the base EVCard API on the provided EContact objects, to avoid extra processing and caching in the EContact code.

Note: We previously considered implementing e_book_get_book_view()'s requested fields parameter, but that changes the whole view, though we only really want to change what is sent for changed/new contacts.

Note: Chris discovered a dedicated e_book_get_contact_uids() API in the Fremantle EDS fork. The new API call was accepted upstream (in master) and [https://bugzilla.gnome.org/show_bug.cgi?id=651446#c11 backported as a patch] (used in the Meego package) for the gnome-2-32 branch, but this is not a full solution.

[https://bugzilla.gnome.org/show_bug.cgi?id=652172 Upstream evolution-data-server bug]

== Optional Parsing of ebook Data in Client ==

The EDS client-side APIs currently parse and (temporarily) store the parsed details of VCards. However, this is often unnecessary because APIs such as e_book_get_book_view() cause it to return the whole VCard anyway. There will be some performance gain from delaying VCard parsing until it is really needed by calls to functions such as e_contact_new_from_vcard().

libebook provides different functionality:
# exchange data as vCard with EDS daemon
# parse items and wrap them in a C API

QtContacts replaces the second part. An obvious performance improvement is to only do the second part if the additional C APIs are really called. The following calls then transfer data without any additional parsing in libebook:
* e_book_get_contact()/asynchronous read/view + e_vcard_to_string()
* e_contact_new_from_vcard() + e_book_add_contact()/e_book_commit_contact()/asynchronous storing

Note that libecal (for iCalendar) can already do this, though libebook can't yet.

TODO:
# add delayed parsing to libebook, triggered by API call which needs parsed data

PROBLEM:
# If delayed parsing fails, how can error be reported?
# Should delayed parsing be done always (e_vcard_to_string())) or only if requested? How would that be configured by a libebook user?

An [http://build.meego.com/package/view_file?file=EBookView-Implement-delayed-vcard-parsing.patch&package=evolution-data-server&project=eds&srcmd5=977d2b885bed818cb616599749407e79 e_book_view_set_parse_vcards() patch], already used in the Meego package, partly implements this already. This patch was taken from an old fremantle EDS patch that was never actually used, and is generally considered to be a bad hack that should not be accepted upstream.

[https://bugzilla.gnome.org/show_bug.cgi?id=652173 Upstream evolution-data-server bug]

== Asynchronous API for storing items in libecal ==

libecal should allow the caller to request storing of data without blocking on processing of the data. It should notify the caller when the data has been stored, or when processing has failed.

These functions may require async versions:
* icalcomponent_new_from_string()
* e_cal_modify_object[_with_mod]
* e_cal_create_object()

Making libical asynchronous is almost impossible without going multithreaded. Let's focus on e_cal D-Bus communication first.

Rob started some work on this in the rbradford-wip-ecal-async-api branch. Perhaps you can reuse some of that work. It is not functional, but might be a source of inspiration or code fragments.

[https://bugzilla.gnome.org/show_bug.cgi?id=652174 Upstream evolution-data-server bug]

== More Efficient Access to Meta Data for Change Tracking ==

This was discussed a while ago: [[http://www.mail-archive.com/evolution-hackers@gnome.org/msg03029.html
[Evolution-hackers] concurrent modifications of items in GUI and EDS database]]

Data synchronization only needs to know which items exist (list of local IDs) and which revision of each item is currently stored. Revisions can be identified with a string which changes for each modification. Restoring an older revision should (but doesn't have to) restore the revision string.

For calendar, UID+RECURRENCE-ID and LAST-MODIFIED can be used. For contacts, UID + REV.

By comparing a stored list against the current one, changes can be detected without relying on journals in the daemon or comparison of time stamps. Journals are hard to maintain (how many of them, how long?). Comparison of time stamps has problems when changes are allowed while a sync runs and assumes that system time is linear (which is not always true).

As an additional use case, local ID + revision string could be used to prevent overwriting stale data in the server, as in HTTP PUT eTag checks.

TODO:

The Maemo Fremantle port of EDS, which is of course already open source, apparently has some solution for this, according to Mathias Hasselman, so that should be investigated first.

However, this should probably be made possible by:

# Extending the EDS query language with a flag that limits the delivered data to just:
## UID and REV for libebook (REV is in VCard 3.0)
## UID+RECURRENCE-ID+LAST-MODIFIED, for libecal
# Implementing these queries for the local contact and calendar backends. The query would be rejected, with an error, for other backends.

== Atomic Updates ==

Later, implement atomic changes as discussed in the mail thread: [[http://www.mail-archive.com/evolution-hackers@gnome.org/msg03029.html

== Quick check for "data changed" ==

The libebook and libical APIs should offer a way to discover if any data whatsoever has changed. This would, for instance, help SyncEvolution to avoid unnecessary detailed checking. This could just check the modification dates of local storage files, such as the BDB .db file, or .ics file, as long as we do not use those files to store permanent caches of parsed data. This would be much quicker than listing items for the whole database and comparing their revisions strings, as is currently necessary.

This would require small API additions, to check that the opaque revision strings are unchanged, such as

gchar* e_book_get_revision(EBook *book)
gboolean e_book_check_revision(EBook *book , const gchar* revision)

and

gchar* e_cal_get_revision(ECal *ecal)
gboolean e_cal_check_revision(ECal ecal, const gchar revision)

The semantic would be this:
* If the revision string is empty or different, then the content of the database '''might''' have changed. A more detailed check is necessary to determine if it has really changed. This will be the case for non-local storage.
* If it is non-empty and the same as before, the nothing has changed. No further check will be necessary.

[https://bugzilla.gnome.org/show_bug.cgi?id=652175 libebook: Upstream evolution-data-server bug] 
[https://bugzilla.gnome.org/show_bug.cgi?id=652177 libecal: Upstream evolution-data-server bug]

== Contacts: Store PHOTO Data as Plain Files ==

Photos should be transferred over D-Bus via a filepath to a local file rather than transferring the actual photo data over D-Bus. The VCard format already allows this option (See the PHOTO property). We must be careful to keep these local files in sync and to remove them when contacts are removed, and to restrict access to the files from other processes.

There should be utility code available to libebook users to convert between the different representations.

Goals:
# Apps should be able to create photo files without ever passing the data to libebook.
# External PHOTO file and corresponding contact must be kept in sync:
## If a contact is deleted, the photo also needs to be removed.
## If a photo file is created and then storing the contact fails, who removes the file?
# Access to photo data must be limited to processes which have the right permissions.

Proposal:
* For each ~/.local/share/evolution/foo calendar, store photos in <photo path>=~/.local/share/evolution/foo/photos, with same permissions.
* New photo files are created with a [http://en.wikipedia.org/wiki/Universally_unique_identifier UUID] as base name and suffix according to the file content.
* If a contact is deleted and has a PHOTO;VALUE=uri:file://<photo path>/*, delete that file.
* For apps which want to create the photo file directly:
** Apps get a file descriptor and the path from a new libebook API.
** They own the file until the EDS server acknowledges the successful storing of a contact with a matching PHOTO;VALUE=uri:file property. If that fails, the app must remove the file.
* Add new API calls to libebook which split out resp. include data inline.
** Must track ownership of file after splitting it out and before storing - might never get stored!

Open:
* Should there be automatic garbage collection of stale photos in case something went wrong and a photo wasn't properly deleted?
* Trust TYPE=JPEG/PNG/... or look into file content to determine type?

== Future: Content Protection ==

Only authorized applications (= processes) with the necessary permissions should be able to read/write data. Exact permissions and mechanisms for verifying them to be defined as part of the (revised?) MeeGo architecture.

This can be added as follows:
# lock down file permissions
# add permission checks to EDS D-Bus servers

Currently not planned. Depends on the future MeeGo security infrastructure.

Architecture/planning/evolution-data-server/eds-improvements

2011-06-09T10:08:43Z

Murrayc: /* Quick check for "data changed" */

This page describes improvements that are desirable or even essential for good performance of Evolution Data Server (EDS) as PIM storage in MeeGo.

All of these changes should be developed for the evolution-data-server gnome-2-32 branch, as used by MeeGo 1.2, then ported to master for review by upstream maintainers. Most changes will not be accepted in upstream's gnome-2-32 branch, because is is stable, but Meego will apply them as patches to its package.

== Change Tracking ==

EDS is based on the model/view/controller principle: an application opens a database (calendar or addressbook) and requests a view with server-side filtering, then the application is sent:
# all data covered by the view
# the "all data sent" signal
# changes (add/update/remove) while the view exists

Applications modify data with asynchronous requests (controller) and keep showing the old data until the change notification comes in. The exact changes made to the data are determined by the server (for example when meeting invitations are sent).

QtContacts and KCalCore do not have such a strict model. With QtContacts the app simply loads and stores contacts and is responsible for maintaining its own view. Changes are sent for changes made after the database was opened. KCalCore is based on the concept of loading a calendar (partially) into the client, modifying it there, then writing it back. Changes are also sent while the storage is open.

TODO:
* https://bugs.meego.com/show_bug.cgi?id=18760 - "QtContacts-EDS + EDS: more flexible EBookView (UID only, only future changes)"

=== Suppress Unnecessary Data Transfer ===

EDS currently sends all data when a view is requested, which is unnecessary when the client is only interested in change notifications. An extension to the EDS query language could optionally suppresses that sending of existing data when the view is opened, to be implemented only for the local contacts and calendar backends. This would allow easier integration with the QContact API.

The Fremantle EDS fork also had a “freezable” (static) capability for libebook, which should probably be investigated. See, for instance: http://maemo.org/api_refs/5.0/5.0-final/libebook/EBookView.html#e-book-view-set-freezable

One possibility for adding the flag is to (ab)use the existing query language with a match for special fields:
* in the client: query = e_book_query_orv(e_book_query_field_exists("X-EDS-QUERY-FLAG-SKIP-DATA-DUMP"), <real query>, NULL);
* in the server: "if (query is OR-Query and first clause is exists("X-EDS-QUERY-FLAG-SKIP-DATA-DUMP")) { skip_data_dump = TRUE; query = second clause; }

[https://bugzilla.gnome.org/show_bug.cgi?id=652171 Upstream evolution-data-server bug]

=== Nofify with UIDs Only ===

EBookView emits signals when contacts are changed, added, or removed. But those signals sends lists of entire EContacts, containing the entire contact data, though we often have all the data already and just want to know which contacts have changed.

[http://developer.gnome.org/libebook/stable/EBook.html#e-book-get-book-view e_book_get_book_view()] takes an [http://developer.gnome.org/libebook/stable/libebook-e-book-query.html#EBookQuery EBookQuery] which could be extended to take a new flag to make the EBookView send only the UID in the signals' EContacts. For instance, "X-EDS-QUERY-FLAG-NOTIFY-UID-ONLY". The UID is already stored separately for use as the primary key of the local-storage backend's BDB database, so no VCard parsing should be necessary to provide it.

This would allow easier integration with the QContact API, greatly improving performance.

Note that users of this feature should probably use just the base EVCard API on the provided EContact objects, to avoid extra processing and caching in the EContact code.

Note: We previously considered implementing e_book_get_book_view()'s requested fields parameter, but that changes the whole view, though we only really want to change what is sent for changed/new contacts.

Note: Chris discovered a dedicated e_book_get_contact_uids() API in the Fremantle EDS fork. The new API call was accepted upstream (in master) and [https://bugzilla.gnome.org/show_bug.cgi?id=651446#c11 backported as a patch] (used in the Meego package) for the gnome-2-32 branch, but this is not a full solution.

[https://bugzilla.gnome.org/show_bug.cgi?id=652172 Upstream evolution-data-server bug]

== Optional Parsing of ebook Data in Client ==

The EDS client-side APIs currently parse and (temporarily) store the parsed details of VCards. However, this is often unnecessary because APIs such as e_book_get_book_view() cause it to return the whole VCard anyway. There will be some performance gain from delaying VCard parsing until it is really needed by calls to functions such as e_contact_new_from_vcard().

libebook provides different functionality:
# exchange data as vCard with EDS daemon
# parse items and wrap them in a C API

QtContacts replaces the second part. An obvious performance improvement is to only do the second part if the additional C APIs are really called. The following calls then transfer data without any additional parsing in libebook:
* e_book_get_contact()/asynchronous read/view + e_vcard_to_string()
* e_contact_new_from_vcard() + e_book_add_contact()/e_book_commit_contact()/asynchronous storing

Note that libecal (for iCalendar) can already do this, though libebook can't yet.

TODO:
# add delayed parsing to libebook, triggered by API call which needs parsed data

PROBLEM:
# If delayed parsing fails, how can error be reported?
# Should delayed parsing be done always (e_vcard_to_string())) or only if requested? How would that be configured by a libebook user?

An [http://build.meego.com/package/view_file?file=EBookView-Implement-delayed-vcard-parsing.patch&package=evolution-data-server&project=eds&srcmd5=977d2b885bed818cb616599749407e79 e_book_view_set_parse_vcards() patch], already used in the Meego package, partly implements this already. This patch was taken from an old fremantle EDS patch that was never actually used, and is generally considered to be a bad hack that should not be accepted upstream.

[https://bugzilla.gnome.org/show_bug.cgi?id=652173 Upstream evolution-data-server bug]

== Asynchronous API for storing items in libecal ==

libecal should allow the caller to request storing of data without blocking on processing of the data. It should notify the caller when the data has been stored, or when processing has failed.

These functions may require async versions:
* icalcomponent_new_from_string()
* e_cal_modify_object[_with_mod]
* e_cal_create_object()

Making libical asynchronous is almost impossible without going multithreaded. Let's focus on e_cal D-Bus communication first.

Rob started some work on this in the rbradford-wip-ecal-async-api branch. Perhaps you can reuse some of that work. It is not functional, but might be a source of inspiration or code fragments.

[https://bugzilla.gnome.org/show_bug.cgi?id=652174 Upstream evolution-data-server bug]

== More Efficient Access to Meta Data for Change Tracking ==

This was discussed a while ago: [[http://www.mail-archive.com/evolution-hackers@gnome.org/msg03029.html
[Evolution-hackers] concurrent modifications of items in GUI and EDS database]]

Data synchronization only needs to know which items exist (list of local IDs) and which revision of each item is currently stored. Revisions can be identified with a string which changes for each modification. Restoring an older revision should (but doesn't have to) restore the revision string.

For calendar, UID+RECURRENCE-ID and LAST-MODIFIED can be used. For contacts, UID + REV.

By comparing a stored list against the current one, changes can be detected without relying on journals in the daemon or comparison of time stamps. Journals are hard to maintain (how many of them, how long?). Comparison of time stamps has problems when changes are allowed while a sync runs and assumes that system time is linear (which is not always true).

As an additional use case, local ID + revision string could be used to prevent overwriting stale data in the server, as in HTTP PUT eTag checks.

TODO:

The Maemo Fremantle port of EDS, which is of course already open source, apparently has some solution for this, according to Mathias Hasselman, so that should be investigated first.

However, this should probably be made possible by:

# Extending the EDS query language with a flag that limits the delivered data to just:
## UID and REV for libebook (REV is in VCard 3.0)
## UID+RECURRENCE-ID+LAST-MODIFIED, for libecal
# Implementing these queries for the local contact and calendar backends. The query would be rejected, with an error, for other backends.

== Atomic Updates ==

Later, implement atomic changes as discussed in the mail thread: [[http://www.mail-archive.com/evolution-hackers@gnome.org/msg03029.html

== Quick check for "data changed" ==

The libebook and libical APIs should offer a way to discover if any data whatsoever has changed. This would, for instance, help SyncEvolution to avoid unnecessary detailed checking. This could just check the modification dates of local storage files, such as the BDB .db file, or .ics file, as long as we do not use those files to store permanent caches of parsed data. This would be much quicker than listing items for the whole database and comparing their revisions strings, as is currently necessary.

This would require small API additions, to check that the opaque revision strings are unchanged, such as

gchar* e_book_get_revision(EBook *book)
gboolean e_book_check_revision(EBook *book , const gchar* revision)

and

gchar* e_cal_get_revision(ECal *ecal)
gboolean e_cal_check_revision(ECal ecal, const gchar revision)

The semantic would be this:
* If the revision string is empty or different, then the content of the database '''might''' have changed. A more detailed check is necessary to determine if it has really changed. This will be the case for non-local storage.
* If it is non-empty and the same as before, the nothing has changed. No further check will be necessary.

[https://bugzilla.gnome.org/show_bug.cgi?id=652175 libebook: Upstream evolution-data-server bug] 
[https://bugzilla.gnome.org/show_bug.cgi?id=652177 libecal: Upstream evolution-data-server bug]

== Contacts: Store PHOTO Data as Plain Files ==

Photos should be transferred over D-Bus via a filepath to a local file rather than transferring the actual photo data over D-Bus. The VCard format already allows this option (See the PHOTO property). We must be careful to keep these local files in sync and to remove them when contacts are removed, and to restrict access to the files from other processes.

There should be utility code available to libebook users to convert between the different representations.

Goals:
# Apps should be able to create photo files without ever passing the data to libebook.
# External PHOTO file and corresponding contact must be kept in sync:
** If a contact is deleted, the photo also needs to be removed.
** If a photo file is created and then storing the contact fails, who removes the file?
# Access to photo data must be limited to processes which have the right permissions.

Proposal:
* For each ~/.local/share/evolution/foo calendar, store photos in <photo path>=~/.local/share/evolution/foo/photos, with same permissions.
* New photo files are created with a [http://en.wikipedia.org/wiki/Universally_unique_identifier UUID] as base name and suffix according to the file content.
* If a contact is deleted and has a PHOTO;VALUE=uri:file://<photo path>/*, delete that file.
* For apps which want to create the photo file directly:
** Apps get a file descriptor and the path from a new libebook API.
** They own the file until the EDS server acknowledges the successful storing of a contact with a matching PHOTO;VALUE=uri:file property. If that fails, the app must remove the file.
* Add new API calls to libebook which split out resp. include data inline.
** Must track ownership of file after splitting it out and before storing - might never get stored!

Open:
* Should there be automatic garbage collection of stale photos in case something went wrong and a photo wasn't properly deleted?
* Trust TYPE=JPEG/PNG/... or look into file content to determine type?

== Future: Content Protection ==

Only authorized applications (= processes) with the necessary permissions should be able to read/write data. Exact permissions and mechanisms for verifying them to be defined as part of the (revised?) MeeGo architecture.

This can be added as follows:
# lock down file permissions
# add permission checks to EDS D-Bus servers

Currently not planned. Depends on the future MeeGo security infrastructure.

Architecture/planning/evolution-data-server/eds-improvements

2011-06-09T10:08:23Z

Murrayc: /* Quick check for "data changed" */

This page describes improvements that are desirable or even essential for good performance of Evolution Data Server (EDS) as PIM storage in MeeGo.

All of these changes should be developed for the evolution-data-server gnome-2-32 branch, as used by MeeGo 1.2, then ported to master for review by upstream maintainers. Most changes will not be accepted in upstream's gnome-2-32 branch, because is is stable, but Meego will apply them as patches to its package.

== Change Tracking ==

EDS is based on the model/view/controller principle: an application opens a database (calendar or addressbook) and requests a view with server-side filtering, then the application is sent:
# all data covered by the view
# the "all data sent" signal
# changes (add/update/remove) while the view exists

Applications modify data with asynchronous requests (controller) and keep showing the old data until the change notification comes in. The exact changes made to the data are determined by the server (for example when meeting invitations are sent).

QtContacts and KCalCore do not have such a strict model. With QtContacts the app simply loads and stores contacts and is responsible for maintaining its own view. Changes are sent for changes made after the database was opened. KCalCore is based on the concept of loading a calendar (partially) into the client, modifying it there, then writing it back. Changes are also sent while the storage is open.

TODO:
* https://bugs.meego.com/show_bug.cgi?id=18760 - "QtContacts-EDS + EDS: more flexible EBookView (UID only, only future changes)"

=== Suppress Unnecessary Data Transfer ===

EDS currently sends all data when a view is requested, which is unnecessary when the client is only interested in change notifications. An extension to the EDS query language could optionally suppresses that sending of existing data when the view is opened, to be implemented only for the local contacts and calendar backends. This would allow easier integration with the QContact API.

The Fremantle EDS fork also had a “freezable” (static) capability for libebook, which should probably be investigated. See, for instance: http://maemo.org/api_refs/5.0/5.0-final/libebook/EBookView.html#e-book-view-set-freezable

One possibility for adding the flag is to (ab)use the existing query language with a match for special fields:
* in the client: query = e_book_query_orv(e_book_query_field_exists("X-EDS-QUERY-FLAG-SKIP-DATA-DUMP"), <real query>, NULL);
* in the server: "if (query is OR-Query and first clause is exists("X-EDS-QUERY-FLAG-SKIP-DATA-DUMP")) { skip_data_dump = TRUE; query = second clause; }

[https://bugzilla.gnome.org/show_bug.cgi?id=652171 Upstream evolution-data-server bug]

=== Nofify with UIDs Only ===

EBookView emits signals when contacts are changed, added, or removed. But those signals sends lists of entire EContacts, containing the entire contact data, though we often have all the data already and just want to know which contacts have changed.

[http://developer.gnome.org/libebook/stable/EBook.html#e-book-get-book-view e_book_get_book_view()] takes an [http://developer.gnome.org/libebook/stable/libebook-e-book-query.html#EBookQuery EBookQuery] which could be extended to take a new flag to make the EBookView send only the UID in the signals' EContacts. For instance, "X-EDS-QUERY-FLAG-NOTIFY-UID-ONLY". The UID is already stored separately for use as the primary key of the local-storage backend's BDB database, so no VCard parsing should be necessary to provide it.

This would allow easier integration with the QContact API, greatly improving performance.

Note that users of this feature should probably use just the base EVCard API on the provided EContact objects, to avoid extra processing and caching in the EContact code.

Note: We previously considered implementing e_book_get_book_view()'s requested fields parameter, but that changes the whole view, though we only really want to change what is sent for changed/new contacts.

Note: Chris discovered a dedicated e_book_get_contact_uids() API in the Fremantle EDS fork. The new API call was accepted upstream (in master) and [https://bugzilla.gnome.org/show_bug.cgi?id=651446#c11 backported as a patch] (used in the Meego package) for the gnome-2-32 branch, but this is not a full solution.

[https://bugzilla.gnome.org/show_bug.cgi?id=652172 Upstream evolution-data-server bug]

== Optional Parsing of ebook Data in Client ==

The EDS client-side APIs currently parse and (temporarily) store the parsed details of VCards. However, this is often unnecessary because APIs such as e_book_get_book_view() cause it to return the whole VCard anyway. There will be some performance gain from delaying VCard parsing until it is really needed by calls to functions such as e_contact_new_from_vcard().

libebook provides different functionality:
# exchange data as vCard with EDS daemon
# parse items and wrap them in a C API

QtContacts replaces the second part. An obvious performance improvement is to only do the second part if the additional C APIs are really called. The following calls then transfer data without any additional parsing in libebook:
* e_book_get_contact()/asynchronous read/view + e_vcard_to_string()
* e_contact_new_from_vcard() + e_book_add_contact()/e_book_commit_contact()/asynchronous storing

Note that libecal (for iCalendar) can already do this, though libebook can't yet.

TODO:
# add delayed parsing to libebook, triggered by API call which needs parsed data

PROBLEM:
# If delayed parsing fails, how can error be reported?
# Should delayed parsing be done always (e_vcard_to_string())) or only if requested? How would that be configured by a libebook user?

An [http://build.meego.com/package/view_file?file=EBookView-Implement-delayed-vcard-parsing.patch&package=evolution-data-server&project=eds&srcmd5=977d2b885bed818cb616599749407e79 e_book_view_set_parse_vcards() patch], already used in the Meego package, partly implements this already. This patch was taken from an old fremantle EDS patch that was never actually used, and is generally considered to be a bad hack that should not be accepted upstream.

[https://bugzilla.gnome.org/show_bug.cgi?id=652173 Upstream evolution-data-server bug]

== Asynchronous API for storing items in libecal ==

libecal should allow the caller to request storing of data without blocking on processing of the data. It should notify the caller when the data has been stored, or when processing has failed.

These functions may require async versions:
* icalcomponent_new_from_string()
* e_cal_modify_object[_with_mod]
* e_cal_create_object()

Making libical asynchronous is almost impossible without going multithreaded. Let's focus on e_cal D-Bus communication first.

Rob started some work on this in the rbradford-wip-ecal-async-api branch. Perhaps you can reuse some of that work. It is not functional, but might be a source of inspiration or code fragments.

[https://bugzilla.gnome.org/show_bug.cgi?id=652174 Upstream evolution-data-server bug]

== More Efficient Access to Meta Data for Change Tracking ==

This was discussed a while ago: [[http://www.mail-archive.com/evolution-hackers@gnome.org/msg03029.html
[Evolution-hackers] concurrent modifications of items in GUI and EDS database]]

Data synchronization only needs to know which items exist (list of local IDs) and which revision of each item is currently stored. Revisions can be identified with a string which changes for each modification. Restoring an older revision should (but doesn't have to) restore the revision string.

For calendar, UID+RECURRENCE-ID and LAST-MODIFIED can be used. For contacts, UID + REV.

By comparing a stored list against the current one, changes can be detected without relying on journals in the daemon or comparison of time stamps. Journals are hard to maintain (how many of them, how long?). Comparison of time stamps has problems when changes are allowed while a sync runs and assumes that system time is linear (which is not always true).

As an additional use case, local ID + revision string could be used to prevent overwriting stale data in the server, as in HTTP PUT eTag checks.

TODO:

The Maemo Fremantle port of EDS, which is of course already open source, apparently has some solution for this, according to Mathias Hasselman, so that should be investigated first.

However, this should probably be made possible by:

# Extending the EDS query language with a flag that limits the delivered data to just:
## UID and REV for libebook (REV is in VCard 3.0)
## UID+RECURRENCE-ID+LAST-MODIFIED, for libecal
# Implementing these queries for the local contact and calendar backends. The query would be rejected, with an error, for other backends.

== Atomic Updates ==

Later, implement atomic changes as discussed in the mail thread: [[http://www.mail-archive.com/evolution-hackers@gnome.org/msg03029.html

== Quick check for "data changed" ==

The libebook and libical APIs should offer a way to discover if any data whatsoever has changed. This would, for instance, help SyncEvolution to avoid unnecessary detailed checking. This could just check the modification dates of local storage files, such as the BDB .db file, or .ics file, as long as we do not use those files to store permanent caches of parsed data. This would be much quicker than listing items for the whole database and comparing their revisions strings, as is currently necessary.

This would require small API additions, to check that the opaque revision strings are unchanged, such as

gchar* e_book_get_revision(EBook *book)
gboolean e_book_check_revision(EBook *book , const gchar* revision)

and

gchar* e_cal_get_revision(ECal *ecal)
gboolean e_cal_check_revision(ECal ecal, const gchar revision)

The semantic would be this:
* If the revision string is empty or different, then the content of the database '''might''' have changed. A more detailed check is necessary to determine if it has really changed. This will be the case for non-local storage.
* If it is non-empty and the same as before, the nothing has changed. No further check will be necessary.

[https://bugzilla.gnome.org/show_bug.cgi?id=652175 libebook: Upstream evolution-data-server bug]
[https://bugzilla.gnome.org/show_bug.cgi?id=652177 libecal: Upstream evolution-data-server bug]

== Contacts: Store PHOTO Data as Plain Files ==

Photos should be transferred over D-Bus via a filepath to a local file rather than transferring the actual photo data over D-Bus. The VCard format already allows this option (See the PHOTO property). We must be careful to keep these local files in sync and to remove them when contacts are removed, and to restrict access to the files from other processes.

There should be utility code available to libebook users to convert between the different representations.

Goals:
# Apps should be able to create photo files without ever passing the data to libebook.
# External PHOTO file and corresponding contact must be kept in sync:
** If a contact is deleted, the photo also needs to be removed.
** If a photo file is created and then storing the contact fails, who removes the file?
# Access to photo data must be limited to processes which have the right permissions.

Proposal:
* For each ~/.local/share/evolution/foo calendar, store photos in <photo path>=~/.local/share/evolution/foo/photos, with same permissions.
* New photo files are created with a [http://en.wikipedia.org/wiki/Universally_unique_identifier UUID] as base name and suffix according to the file content.
* If a contact is deleted and has a PHOTO;VALUE=uri:file://<photo path>/*, delete that file.
* For apps which want to create the photo file directly:
** Apps get a file descriptor and the path from a new libebook API.
** They own the file until the EDS server acknowledges the successful storing of a contact with a matching PHOTO;VALUE=uri:file property. If that fails, the app must remove the file.
* Add new API calls to libebook which split out resp. include data inline.
** Must track ownership of file after splitting it out and before storing - might never get stored!

Open:
* Should there be automatic garbage collection of stale photos in case something went wrong and a photo wasn't properly deleted?
* Trust TYPE=JPEG/PNG/... or look into file content to determine type?

== Future: Content Protection ==

Only authorized applications (= processes) with the necessary permissions should be able to read/write data. Exact permissions and mechanisms for verifying them to be defined as part of the (revised?) MeeGo architecture.

This can be added as follows:
# lock down file permissions
# add permission checks to EDS D-Bus servers

Currently not planned. Depends on the future MeeGo security infrastructure.

Architecture/planning/evolution-data-server/eds-improvements

2011-06-09T10:02:15Z

Murrayc: /* Optional Parsing of ebook Data in Client */

This page describes improvements that are desirable or even essential for good performance of Evolution Data Server (EDS) as PIM storage in MeeGo.

All of these changes should be developed for the evolution-data-server gnome-2-32 branch, as used by MeeGo 1.2, then ported to master for review by upstream maintainers. Most changes will not be accepted in upstream's gnome-2-32 branch, because is is stable, but Meego will apply them as patches to its package.

== Change Tracking ==

EDS is based on the model/view/controller principle: an application opens a database (calendar or addressbook) and requests a view with server-side filtering, then the application is sent:
# all data covered by the view
# the "all data sent" signal
# changes (add/update/remove) while the view exists

Applications modify data with asynchronous requests (controller) and keep showing the old data until the change notification comes in. The exact changes made to the data are determined by the server (for example when meeting invitations are sent).

QtContacts and KCalCore do not have such a strict model. With QtContacts the app simply loads and stores contacts and is responsible for maintaining its own view. Changes are sent for changes made after the database was opened. KCalCore is based on the concept of loading a calendar (partially) into the client, modifying it there, then writing it back. Changes are also sent while the storage is open.

TODO:
* https://bugs.meego.com/show_bug.cgi?id=18760 - "QtContacts-EDS + EDS: more flexible EBookView (UID only, only future changes)"

=== Suppress Unnecessary Data Transfer ===

EDS currently sends all data when a view is requested, which is unnecessary when the client is only interested in change notifications. An extension to the EDS query language could optionally suppresses that sending of existing data when the view is opened, to be implemented only for the local contacts and calendar backends. This would allow easier integration with the QContact API.

The Fremantle EDS fork also had a “freezable” (static) capability for libebook, which should probably be investigated. See, for instance: http://maemo.org/api_refs/5.0/5.0-final/libebook/EBookView.html#e-book-view-set-freezable

One possibility for adding the flag is to (ab)use the existing query language with a match for special fields:
* in the client: query = e_book_query_orv(e_book_query_field_exists("X-EDS-QUERY-FLAG-SKIP-DATA-DUMP"), <real query>, NULL);
* in the server: "if (query is OR-Query and first clause is exists("X-EDS-QUERY-FLAG-SKIP-DATA-DUMP")) { skip_data_dump = TRUE; query = second clause; }

[https://bugzilla.gnome.org/show_bug.cgi?id=652171 Upstream evolution-data-server bug]

=== Nofify with UIDs Only ===

EBookView emits signals when contacts are changed, added, or removed. But those signals sends lists of entire EContacts, containing the entire contact data, though we often have all the data already and just want to know which contacts have changed.

[http://developer.gnome.org/libebook/stable/EBook.html#e-book-get-book-view e_book_get_book_view()] takes an [http://developer.gnome.org/libebook/stable/libebook-e-book-query.html#EBookQuery EBookQuery] which could be extended to take a new flag to make the EBookView send only the UID in the signals' EContacts. For instance, "X-EDS-QUERY-FLAG-NOTIFY-UID-ONLY". The UID is already stored separately for use as the primary key of the local-storage backend's BDB database, so no VCard parsing should be necessary to provide it.

This would allow easier integration with the QContact API, greatly improving performance.

Note that users of this feature should probably use just the base EVCard API on the provided EContact objects, to avoid extra processing and caching in the EContact code.

Note: We previously considered implementing e_book_get_book_view()'s requested fields parameter, but that changes the whole view, though we only really want to change what is sent for changed/new contacts.

Note: Chris discovered a dedicated e_book_get_contact_uids() API in the Fremantle EDS fork. The new API call was accepted upstream (in master) and [https://bugzilla.gnome.org/show_bug.cgi?id=651446#c11 backported as a patch] (used in the Meego package) for the gnome-2-32 branch, but this is not a full solution.

[https://bugzilla.gnome.org/show_bug.cgi?id=652172 Upstream evolution-data-server bug]

== Optional Parsing of ebook Data in Client ==

The EDS client-side APIs currently parse and (temporarily) store the parsed details of VCards. However, this is often unnecessary because APIs such as e_book_get_book_view() cause it to return the whole VCard anyway. There will be some performance gain from delaying VCard parsing until it is really needed by calls to functions such as e_contact_new_from_vcard().

libebook provides different functionality:
# exchange data as vCard with EDS daemon
# parse items and wrap them in a C API

QtContacts replaces the second part. An obvious performance improvement is to only do the second part if the additional C APIs are really called. The following calls then transfer data without any additional parsing in libebook:
* e_book_get_contact()/asynchronous read/view + e_vcard_to_string()
* e_contact_new_from_vcard() + e_book_add_contact()/e_book_commit_contact()/asynchronous storing

Note that libecal (for iCalendar) can already do this, though libebook can't yet.

TODO:
# add delayed parsing to libebook, triggered by API call which needs parsed data

PROBLEM:
# If delayed parsing fails, how can error be reported?
# Should delayed parsing be done always (e_vcard_to_string())) or only if requested? How would that be configured by a libebook user?

An [http://build.meego.com/package/view_file?file=EBookView-Implement-delayed-vcard-parsing.patch&package=evolution-data-server&project=eds&srcmd5=977d2b885bed818cb616599749407e79 e_book_view_set_parse_vcards() patch], already used in the Meego package, partly implements this already. This patch was taken from an old fremantle EDS patch that was never actually used, and is generally considered to be a bad hack that should not be accepted upstream.

[https://bugzilla.gnome.org/show_bug.cgi?id=652173 Upstream evolution-data-server bug]

== Asynchronous API for storing items in libecal ==

libecal should allow the caller to request storing of data without blocking on processing of the data. It should notify the caller when the data has been stored, or when processing has failed.

These functions may require async versions:
* icalcomponent_new_from_string()
* e_cal_modify_object[_with_mod]
* e_cal_create_object()

Making libical asynchronous is almost impossible without going multithreaded. Let's focus on e_cal D-Bus communication first.

Rob started some work on this in the rbradford-wip-ecal-async-api branch. Perhaps you can reuse some of that work. It is not functional, but might be a source of inspiration or code fragments.

[https://bugzilla.gnome.org/show_bug.cgi?id=652174 Upstream evolution-data-server bug]

== More Efficient Access to Meta Data for Change Tracking ==

This was discussed a while ago: [[http://www.mail-archive.com/evolution-hackers@gnome.org/msg03029.html
[Evolution-hackers] concurrent modifications of items in GUI and EDS database]]

Data synchronization only needs to know which items exist (list of local IDs) and which revision of each item is currently stored. Revisions can be identified with a string which changes for each modification. Restoring an older revision should (but doesn't have to) restore the revision string.

For calendar, UID+RECURRENCE-ID and LAST-MODIFIED can be used. For contacts, UID + REV.

By comparing a stored list against the current one, changes can be detected without relying on journals in the daemon or comparison of time stamps. Journals are hard to maintain (how many of them, how long?). Comparison of time stamps has problems when changes are allowed while a sync runs and assumes that system time is linear (which is not always true).

As an additional use case, local ID + revision string could be used to prevent overwriting stale data in the server, as in HTTP PUT eTag checks.

TODO:

The Maemo Fremantle port of EDS, which is of course already open source, apparently has some solution for this, according to Mathias Hasselman, so that should be investigated first.

However, this should probably be made possible by:

# Extending the EDS query language with a flag that limits the delivered data to just:
## UID and REV for libebook (REV is in VCard 3.0)
## UID+RECURRENCE-ID+LAST-MODIFIED, for libecal
# Implementing these queries for the local contact and calendar backends. The query would be rejected, with an error, for other backends.

== Atomic Updates ==

Later, implement atomic changes as discussed in the mail thread: [[http://www.mail-archive.com/evolution-hackers@gnome.org/msg03029.html

== Quick check for "data changed" ==

The libebook and libical APIs should offer a way to discover if any data whatsoever has changed. This would, for instance, help SyncEvolution to avoid unnecessary detailed checking. This could just check the modification dates of local storage files, such as the BDB .db file, or .ics file, as long as we do not use those files to store permanent caches of parsed data. This would be much quicker than listing items for the whole database and comparing their revisions strings, as is currently necessary.

This would require small API additions, to check that the opaque revision strings are unchanged, such as

gchar* e_book_get_revision(EBook *book)
gboolean e_book_check_revision(EBook *book , const gchar* revision)

and

gchar* e_cal_get_revision(ECal *ecal)
gboolean e_cal_check_revision(ECal ecal, const gchar revision)

The semantic would be this:
* If the revision string is empty or different, then the content of the database '''might''' have changed. A more detailed check is necessary to determine if it has really changed. This will be the case for non-local storage.
* If it is non-empty and the same as before, the nothing has changed. No further check will be necessary.

== Contacts: Store PHOTO Data as Plain Files ==

Photos should be transferred over D-Bus via a filepath to a local file rather than transferring the actual photo data over D-Bus. The VCard format already allows this option (See the PHOTO property). We must be careful to keep these local files in sync and to remove them when contacts are removed, and to restrict access to the files from other processes.

There should be utility code available to libebook users to convert between the different representations.

Goals:
# Apps should be able to create photo files without ever passing the data to libebook.
# External PHOTO file and corresponding contact must be kept in sync:
** If a contact is deleted, the photo also needs to be removed.
** If a photo file is created and then storing the contact fails, who removes the file?
# Access to photo data must be limited to processes which have the right permissions.

Proposal:
* For each ~/.local/share/evolution/foo calendar, store photos in <photo path>=~/.local/share/evolution/foo/photos, with same permissions.
* New photo files are created with a [http://en.wikipedia.org/wiki/Universally_unique_identifier UUID] as base name and suffix according to the file content.
* If a contact is deleted and has a PHOTO;VALUE=uri:file://<photo path>/*, delete that file.
* For apps which want to create the photo file directly:
** Apps get a file descriptor and the path from a new libebook API.
** They own the file until the EDS server acknowledges the successful storing of a contact with a matching PHOTO;VALUE=uri:file property. If that fails, the app must remove the file.
* Add new API calls to libebook which split out resp. include data inline.
** Must track ownership of file after splitting it out and before storing - might never get stored!

Open:
* Should there be automatic garbage collection of stale photos in case something went wrong and a photo wasn't properly deleted?
* Trust TYPE=JPEG/PNG/... or look into file content to determine type?

== Future: Content Protection ==

Only authorized applications (= processes) with the necessary permissions should be able to read/write data. Exact permissions and mechanisms for verifying them to be defined as part of the (revised?) MeeGo architecture.

This can be added as follows:
# lock down file permissions
# add permission checks to EDS D-Bus servers

Currently not planned. Depends on the future MeeGo security infrastructure.

Architecture/planning/evolution-data-server/eds-improvements

2011-06-09T10:01:35Z

Murrayc: /* Asynchronous API for storing items in libecal */

This page describes improvements that are desirable or even essential for good performance of Evolution Data Server (EDS) as PIM storage in MeeGo.

All of these changes should be developed for the evolution-data-server gnome-2-32 branch, as used by MeeGo 1.2, then ported to master for review by upstream maintainers. Most changes will not be accepted in upstream's gnome-2-32 branch, because is is stable, but Meego will apply them as patches to its package.

== Change Tracking ==

EDS is based on the model/view/controller principle: an application opens a database (calendar or addressbook) and requests a view with server-side filtering, then the application is sent:
# all data covered by the view
# the "all data sent" signal
# changes (add/update/remove) while the view exists

Applications modify data with asynchronous requests (controller) and keep showing the old data until the change notification comes in. The exact changes made to the data are determined by the server (for example when meeting invitations are sent).

QtContacts and KCalCore do not have such a strict model. With QtContacts the app simply loads and stores contacts and is responsible for maintaining its own view. Changes are sent for changes made after the database was opened. KCalCore is based on the concept of loading a calendar (partially) into the client, modifying it there, then writing it back. Changes are also sent while the storage is open.

TODO:
* https://bugs.meego.com/show_bug.cgi?id=18760 - "QtContacts-EDS + EDS: more flexible EBookView (UID only, only future changes)"

=== Suppress Unnecessary Data Transfer ===

EDS currently sends all data when a view is requested, which is unnecessary when the client is only interested in change notifications. An extension to the EDS query language could optionally suppresses that sending of existing data when the view is opened, to be implemented only for the local contacts and calendar backends. This would allow easier integration with the QContact API.

The Fremantle EDS fork also had a “freezable” (static) capability for libebook, which should probably be investigated. See, for instance: http://maemo.org/api_refs/5.0/5.0-final/libebook/EBookView.html#e-book-view-set-freezable

One possibility for adding the flag is to (ab)use the existing query language with a match for special fields:
* in the client: query = e_book_query_orv(e_book_query_field_exists("X-EDS-QUERY-FLAG-SKIP-DATA-DUMP"), <real query>, NULL);
* in the server: "if (query is OR-Query and first clause is exists("X-EDS-QUERY-FLAG-SKIP-DATA-DUMP")) { skip_data_dump = TRUE; query = second clause; }

[https://bugzilla.gnome.org/show_bug.cgi?id=652171 Upstream evolution-data-server bug]

=== Nofify with UIDs Only ===

EBookView emits signals when contacts are changed, added, or removed. But those signals sends lists of entire EContacts, containing the entire contact data, though we often have all the data already and just want to know which contacts have changed.

[http://developer.gnome.org/libebook/stable/EBook.html#e-book-get-book-view e_book_get_book_view()] takes an [http://developer.gnome.org/libebook/stable/libebook-e-book-query.html#EBookQuery EBookQuery] which could be extended to take a new flag to make the EBookView send only the UID in the signals' EContacts. For instance, "X-EDS-QUERY-FLAG-NOTIFY-UID-ONLY". The UID is already stored separately for use as the primary key of the local-storage backend's BDB database, so no VCard parsing should be necessary to provide it.

This would allow easier integration with the QContact API, greatly improving performance.

Note that users of this feature should probably use just the base EVCard API on the provided EContact objects, to avoid extra processing and caching in the EContact code.

Note: We previously considered implementing e_book_get_book_view()'s requested fields parameter, but that changes the whole view, though we only really want to change what is sent for changed/new contacts.

Note: Chris discovered a dedicated e_book_get_contact_uids() API in the Fremantle EDS fork. The new API call was accepted upstream (in master) and [https://bugzilla.gnome.org/show_bug.cgi?id=651446#c11 backported as a patch] (used in the Meego package) for the gnome-2-32 branch, but this is not a full solution.

[https://bugzilla.gnome.org/show_bug.cgi?id=652172 Upstream evolution-data-server bug]

== Optional Parsing of ebook Data in Client ==

The EDS client-side APIs currently parse and (temporarily) store the parsed details of VCards. However, this is often unnecessary because APIs such as e_book_get_book_view() cause it to return the whole VCard anyway. There will be some performance gain from delaying VCard parsing until it is really needed by calls to functions such as e_contact_new_from_vcard().

libebook provides different functionality:
# exchange data as vCard with EDS daemon
# parse items and wrap them in a C API

QtContacts replaces the second part. An obvious performance improvement is to only do the second part if the additional C APIs are really called. The following calls then transfer data without any additional parsing in libebook:
* e_book_get_contact()/asynchronous read/view + e_vcard_to_string()
* e_contact_new_from_vcard() + e_book_add_contact()/e_book_commit_contact()/asynchronous storing

Note that libecal (for iCalendar) can already do this, though libebook can't yet.

TODO:
# add delayed parsing to libebook, triggered by API call which needs parsed data

PROBLEM:
# If delayed parsing fails, how can error be reported?
# Should delayed parsing be done always (e_vcard_to_string())) or only if requested? How would that be configured by a libebook user?

An [http://build.meego.com/package/view_file?file=EBookView-Implement-delayed-vcard-parsing.patch&package=evolution-data-server&project=eds&srcmd5=977d2b885bed818cb616599749407e79 e_book_view_set_parse_vcards() patch], already used in the Meego package, partly implements this already. This patch was taken from an old fremantle EDS patch that was never actually used, and is generally considered to be a bad hack that should not be accepted upstream.

[Upstream evolution-data-server bug]

== Asynchronous API for storing items in libecal ==

libecal should allow the caller to request storing of data without blocking on processing of the data. It should notify the caller when the data has been stored, or when processing has failed.

These functions may require async versions:
* icalcomponent_new_from_string()
* e_cal_modify_object[_with_mod]
* e_cal_create_object()

Making libical asynchronous is almost impossible without going multithreaded. Let's focus on e_cal D-Bus communication first.

Rob started some work on this in the rbradford-wip-ecal-async-api branch. Perhaps you can reuse some of that work. It is not functional, but might be a source of inspiration or code fragments.

[https://bugzilla.gnome.org/show_bug.cgi?id=652174 Upstream evolution-data-server bug]

== More Efficient Access to Meta Data for Change Tracking ==

This was discussed a while ago: [[http://www.mail-archive.com/evolution-hackers@gnome.org/msg03029.html
[Evolution-hackers] concurrent modifications of items in GUI and EDS database]]

Data synchronization only needs to know which items exist (list of local IDs) and which revision of each item is currently stored. Revisions can be identified with a string which changes for each modification. Restoring an older revision should (but doesn't have to) restore the revision string.

For calendar, UID+RECURRENCE-ID and LAST-MODIFIED can be used. For contacts, UID + REV.

By comparing a stored list against the current one, changes can be detected without relying on journals in the daemon or comparison of time stamps. Journals are hard to maintain (how many of them, how long?). Comparison of time stamps has problems when changes are allowed while a sync runs and assumes that system time is linear (which is not always true).

As an additional use case, local ID + revision string could be used to prevent overwriting stale data in the server, as in HTTP PUT eTag checks.

TODO:

The Maemo Fremantle port of EDS, which is of course already open source, apparently has some solution for this, according to Mathias Hasselman, so that should be investigated first.

However, this should probably be made possible by:

# Extending the EDS query language with a flag that limits the delivered data to just:
## UID and REV for libebook (REV is in VCard 3.0)
## UID+RECURRENCE-ID+LAST-MODIFIED, for libecal
# Implementing these queries for the local contact and calendar backends. The query would be rejected, with an error, for other backends.

== Atomic Updates ==

Later, implement atomic changes as discussed in the mail thread: [[http://www.mail-archive.com/evolution-hackers@gnome.org/msg03029.html

== Quick check for "data changed" ==

The libebook and libical APIs should offer a way to discover if any data whatsoever has changed. This would, for instance, help SyncEvolution to avoid unnecessary detailed checking. This could just check the modification dates of local storage files, such as the BDB .db file, or .ics file, as long as we do not use those files to store permanent caches of parsed data. This would be much quicker than listing items for the whole database and comparing their revisions strings, as is currently necessary.

This would require small API additions, to check that the opaque revision strings are unchanged, such as

gchar* e_book_get_revision(EBook *book)
gboolean e_book_check_revision(EBook *book , const gchar* revision)

and

gchar* e_cal_get_revision(ECal *ecal)
gboolean e_cal_check_revision(ECal ecal, const gchar revision)

The semantic would be this:
* If the revision string is empty or different, then the content of the database '''might''' have changed. A more detailed check is necessary to determine if it has really changed. This will be the case for non-local storage.
* If it is non-empty and the same as before, the nothing has changed. No further check will be necessary.

== Contacts: Store PHOTO Data as Plain Files ==

Photos should be transferred over D-Bus via a filepath to a local file rather than transferring the actual photo data over D-Bus. The VCard format already allows this option (See the PHOTO property). We must be careful to keep these local files in sync and to remove them when contacts are removed, and to restrict access to the files from other processes.

There should be utility code available to libebook users to convert between the different representations.

Goals:
# Apps should be able to create photo files without ever passing the data to libebook.
# External PHOTO file and corresponding contact must be kept in sync:
** If a contact is deleted, the photo also needs to be removed.
** If a photo file is created and then storing the contact fails, who removes the file?
# Access to photo data must be limited to processes which have the right permissions.

Proposal:
* For each ~/.local/share/evolution/foo calendar, store photos in <photo path>=~/.local/share/evolution/foo/photos, with same permissions.
* New photo files are created with a [http://en.wikipedia.org/wiki/Universally_unique_identifier UUID] as base name and suffix according to the file content.
* If a contact is deleted and has a PHOTO;VALUE=uri:file://<photo path>/*, delete that file.
* For apps which want to create the photo file directly:
** Apps get a file descriptor and the path from a new libebook API.
** They own the file until the EDS server acknowledges the successful storing of a contact with a matching PHOTO;VALUE=uri:file property. If that fails, the app must remove the file.
* Add new API calls to libebook which split out resp. include data inline.
** Must track ownership of file after splitting it out and before storing - might never get stored!

Open:
* Should there be automatic garbage collection of stale photos in case something went wrong and a photo wasn't properly deleted?
* Trust TYPE=JPEG/PNG/... or look into file content to determine type?

== Future: Content Protection ==

Only authorized applications (= processes) with the necessary permissions should be able to read/write data. Exact permissions and mechanisms for verifying them to be defined as part of the (revised?) MeeGo architecture.

This can be added as follows:
# lock down file permissions
# add permission checks to EDS D-Bus servers

Currently not planned. Depends on the future MeeGo security infrastructure.

Architecture/planning/evolution-data-server/eds-improvements

2011-06-09T09:59:03Z

Murrayc: /* Optional Parsing of ebook Data in Client */

This page describes improvements that are desirable or even essential for good performance of Evolution Data Server (EDS) as PIM storage in MeeGo.

All of these changes should be developed for the evolution-data-server gnome-2-32 branch, as used by MeeGo 1.2, then ported to master for review by upstream maintainers. Most changes will not be accepted in upstream's gnome-2-32 branch, because is is stable, but Meego will apply them as patches to its package.

== Change Tracking ==

EDS is based on the model/view/controller principle: an application opens a database (calendar or addressbook) and requests a view with server-side filtering, then the application is sent:
# all data covered by the view
# the "all data sent" signal
# changes (add/update/remove) while the view exists

Applications modify data with asynchronous requests (controller) and keep showing the old data until the change notification comes in. The exact changes made to the data are determined by the server (for example when meeting invitations are sent).

QtContacts and KCalCore do not have such a strict model. With QtContacts the app simply loads and stores contacts and is responsible for maintaining its own view. Changes are sent for changes made after the database was opened. KCalCore is based on the concept of loading a calendar (partially) into the client, modifying it there, then writing it back. Changes are also sent while the storage is open.

TODO:
* https://bugs.meego.com/show_bug.cgi?id=18760 - "QtContacts-EDS + EDS: more flexible EBookView (UID only, only future changes)"

=== Suppress Unnecessary Data Transfer ===

EDS currently sends all data when a view is requested, which is unnecessary when the client is only interested in change notifications. An extension to the EDS query language could optionally suppresses that sending of existing data when the view is opened, to be implemented only for the local contacts and calendar backends. This would allow easier integration with the QContact API.

The Fremantle EDS fork also had a “freezable” (static) capability for libebook, which should probably be investigated. See, for instance: http://maemo.org/api_refs/5.0/5.0-final/libebook/EBookView.html#e-book-view-set-freezable

One possibility for adding the flag is to (ab)use the existing query language with a match for special fields:
* in the client: query = e_book_query_orv(e_book_query_field_exists("X-EDS-QUERY-FLAG-SKIP-DATA-DUMP"), <real query>, NULL);
* in the server: "if (query is OR-Query and first clause is exists("X-EDS-QUERY-FLAG-SKIP-DATA-DUMP")) { skip_data_dump = TRUE; query = second clause; }

[https://bugzilla.gnome.org/show_bug.cgi?id=652171 Upstream evolution-data-server bug]

=== Nofify with UIDs Only ===

EBookView emits signals when contacts are changed, added, or removed. But those signals sends lists of entire EContacts, containing the entire contact data, though we often have all the data already and just want to know which contacts have changed.

[http://developer.gnome.org/libebook/stable/EBook.html#e-book-get-book-view e_book_get_book_view()] takes an [http://developer.gnome.org/libebook/stable/libebook-e-book-query.html#EBookQuery EBookQuery] which could be extended to take a new flag to make the EBookView send only the UID in the signals' EContacts. For instance, "X-EDS-QUERY-FLAG-NOTIFY-UID-ONLY". The UID is already stored separately for use as the primary key of the local-storage backend's BDB database, so no VCard parsing should be necessary to provide it.

This would allow easier integration with the QContact API, greatly improving performance.

Note that users of this feature should probably use just the base EVCard API on the provided EContact objects, to avoid extra processing and caching in the EContact code.

Note: We previously considered implementing e_book_get_book_view()'s requested fields parameter, but that changes the whole view, though we only really want to change what is sent for changed/new contacts.

Note: Chris discovered a dedicated e_book_get_contact_uids() API in the Fremantle EDS fork. The new API call was accepted upstream (in master) and [https://bugzilla.gnome.org/show_bug.cgi?id=651446#c11 backported as a patch] (used in the Meego package) for the gnome-2-32 branch, but this is not a full solution.

[https://bugzilla.gnome.org/show_bug.cgi?id=652172 Upstream evolution-data-server bug]

== Optional Parsing of ebook Data in Client ==

The EDS client-side APIs currently parse and (temporarily) store the parsed details of VCards. However, this is often unnecessary because APIs such as e_book_get_book_view() cause it to return the whole VCard anyway. There will be some performance gain from delaying VCard parsing until it is really needed by calls to functions such as e_contact_new_from_vcard().

libebook provides different functionality:
# exchange data as vCard with EDS daemon
# parse items and wrap them in a C API

QtContacts replaces the second part. An obvious performance improvement is to only do the second part if the additional C APIs are really called. The following calls then transfer data without any additional parsing in libebook:
* e_book_get_contact()/asynchronous read/view + e_vcard_to_string()
* e_contact_new_from_vcard() + e_book_add_contact()/e_book_commit_contact()/asynchronous storing

Note that libecal (for iCalendar) can already do this, though libebook can't yet.

TODO:
# add delayed parsing to libebook, triggered by API call which needs parsed data

PROBLEM:
# If delayed parsing fails, how can error be reported?
# Should delayed parsing be done always (e_vcard_to_string())) or only if requested? How would that be configured by a libebook user?

An [http://build.meego.com/package/view_file?file=EBookView-Implement-delayed-vcard-parsing.patch&package=evolution-data-server&project=eds&srcmd5=977d2b885bed818cb616599749407e79 e_book_view_set_parse_vcards() patch], already used in the Meego package, partly implements this already. This patch was taken from an old fremantle EDS patch that was never actually used, and is generally considered to be a bad hack that should not be accepted upstream.

[Upstream evolution-data-server bug]

== Asynchronous API for storing items in libecal ==

libecal should allow the caller to request storing of data without blocking on processing of the data. It should notify the caller when the data has been stored, or when processing has failed.

These functions may require async versions:
* icalcomponent_new_from_string()
* e_cal_modify_object[_with_mod]
* e_cal_create_object()

Making libical asynchronous is almost impossible without going multithreaded. Let's focus on e_cal D-Bus communication first.

Rob started some work on this in the rbradford-wip-ecal-async-api branch. Perhaps you can reuse some of that work. It is not functional, but might be a source of inspiration or code fragments.

== More Efficient Access to Meta Data for Change Tracking ==

This was discussed a while ago: [[http://www.mail-archive.com/evolution-hackers@gnome.org/msg03029.html
[Evolution-hackers] concurrent modifications of items in GUI and EDS database]]

Data synchronization only needs to know which items exist (list of local IDs) and which revision of each item is currently stored. Revisions can be identified with a string which changes for each modification. Restoring an older revision should (but doesn't have to) restore the revision string.

For calendar, UID+RECURRENCE-ID and LAST-MODIFIED can be used. For contacts, UID + REV.

By comparing a stored list against the current one, changes can be detected without relying on journals in the daemon or comparison of time stamps. Journals are hard to maintain (how many of them, how long?). Comparison of time stamps has problems when changes are allowed while a sync runs and assumes that system time is linear (which is not always true).

As an additional use case, local ID + revision string could be used to prevent overwriting stale data in the server, as in HTTP PUT eTag checks.

TODO:

The Maemo Fremantle port of EDS, which is of course already open source, apparently has some solution for this, according to Mathias Hasselman, so that should be investigated first.

However, this should probably be made possible by:

# Extending the EDS query language with a flag that limits the delivered data to just:
## UID and REV for libebook (REV is in VCard 3.0)
## UID+RECURRENCE-ID+LAST-MODIFIED, for libecal
# Implementing these queries for the local contact and calendar backends. The query would be rejected, with an error, for other backends.

== Atomic Updates ==

Later, implement atomic changes as discussed in the mail thread: [[http://www.mail-archive.com/evolution-hackers@gnome.org/msg03029.html

== Quick check for "data changed" ==

The libebook and libical APIs should offer a way to discover if any data whatsoever has changed. This would, for instance, help SyncEvolution to avoid unnecessary detailed checking. This could just check the modification dates of local storage files, such as the BDB .db file, or .ics file, as long as we do not use those files to store permanent caches of parsed data. This would be much quicker than listing items for the whole database and comparing their revisions strings, as is currently necessary.

This would require small API additions, to check that the opaque revision strings are unchanged, such as

gchar* e_book_get_revision(EBook *book)
gboolean e_book_check_revision(EBook *book , const gchar* revision)

and

gchar* e_cal_get_revision(ECal *ecal)
gboolean e_cal_check_revision(ECal ecal, const gchar revision)

The semantic would be this:
* If the revision string is empty or different, then the content of the database '''might''' have changed. A more detailed check is necessary to determine if it has really changed. This will be the case for non-local storage.
* If it is non-empty and the same as before, the nothing has changed. No further check will be necessary.

== Contacts: Store PHOTO Data as Plain Files ==

Photos should be transferred over D-Bus via a filepath to a local file rather than transferring the actual photo data over D-Bus. The VCard format already allows this option (See the PHOTO property). We must be careful to keep these local files in sync and to remove them when contacts are removed, and to restrict access to the files from other processes.

There should be utility code available to libebook users to convert between the different representations.

Goals:
# Apps should be able to create photo files without ever passing the data to libebook.
# External PHOTO file and corresponding contact must be kept in sync:
** If a contact is deleted, the photo also needs to be removed.
** If a photo file is created and then storing the contact fails, who removes the file?
# Access to photo data must be limited to processes which have the right permissions.

Proposal:
* For each ~/.local/share/evolution/foo calendar, store photos in <photo path>=~/.local/share/evolution/foo/photos, with same permissions.
* New photo files are created with a [http://en.wikipedia.org/wiki/Universally_unique_identifier UUID] as base name and suffix according to the file content.
* If a contact is deleted and has a PHOTO;VALUE=uri:file://<photo path>/*, delete that file.
* For apps which want to create the photo file directly:
** Apps get a file descriptor and the path from a new libebook API.
** They own the file until the EDS server acknowledges the successful storing of a contact with a matching PHOTO;VALUE=uri:file property. If that fails, the app must remove the file.
* Add new API calls to libebook which split out resp. include data inline.
** Must track ownership of file after splitting it out and before storing - might never get stored!

Open:
* Should there be automatic garbage collection of stale photos in case something went wrong and a photo wasn't properly deleted?
* Trust TYPE=JPEG/PNG/... or look into file content to determine type?

== Future: Content Protection ==

Only authorized applications (= processes) with the necessary permissions should be able to read/write data. Exact permissions and mechanisms for verifying them to be defined as part of the (revised?) MeeGo architecture.

This can be added as follows:
# lock down file permissions
# add permission checks to EDS D-Bus servers

Currently not planned. Depends on the future MeeGo security infrastructure.

Architecture/planning/evolution-data-server/eds-improvements

2011-06-09T09:48:48Z

Murrayc: /* Optional Parsing of ebook Data in Client */

This page describes improvements that are desirable or even essential for good performance of Evolution Data Server (EDS) as PIM storage in MeeGo.

All of these changes should be developed for the evolution-data-server gnome-2-32 branch, as used by MeeGo 1.2, then ported to master for review by upstream maintainers. Most changes will not be accepted in upstream's gnome-2-32 branch, because is is stable, but Meego will apply them as patches to its package.

== Change Tracking ==

EDS is based on the model/view/controller principle: an application opens a database (calendar or addressbook) and requests a view with server-side filtering, then the application is sent:
# all data covered by the view
# the "all data sent" signal
# changes (add/update/remove) while the view exists

Applications modify data with asynchronous requests (controller) and keep showing the old data until the change notification comes in. The exact changes made to the data are determined by the server (for example when meeting invitations are sent).

QtContacts and KCalCore do not have such a strict model. With QtContacts the app simply loads and stores contacts and is responsible for maintaining its own view. Changes are sent for changes made after the database was opened. KCalCore is based on the concept of loading a calendar (partially) into the client, modifying it there, then writing it back. Changes are also sent while the storage is open.

TODO:
* https://bugs.meego.com/show_bug.cgi?id=18760 - "QtContacts-EDS + EDS: more flexible EBookView (UID only, only future changes)"

=== Suppress Unnecessary Data Transfer ===

EDS currently sends all data when a view is requested, which is unnecessary when the client is only interested in change notifications. An extension to the EDS query language could optionally suppresses that sending of existing data when the view is opened, to be implemented only for the local contacts and calendar backends. This would allow easier integration with the QContact API.

The Fremantle EDS fork also had a “freezable” (static) capability for libebook, which should probably be investigated. See, for instance: http://maemo.org/api_refs/5.0/5.0-final/libebook/EBookView.html#e-book-view-set-freezable

One possibility for adding the flag is to (ab)use the existing query language with a match for special fields:
* in the client: query = e_book_query_orv(e_book_query_field_exists("X-EDS-QUERY-FLAG-SKIP-DATA-DUMP"), <real query>, NULL);
* in the server: "if (query is OR-Query and first clause is exists("X-EDS-QUERY-FLAG-SKIP-DATA-DUMP")) { skip_data_dump = TRUE; query = second clause; }

[https://bugzilla.gnome.org/show_bug.cgi?id=652171 Upstream evolution-data-server bug]

=== Nofify with UIDs Only ===

EBookView emits signals when contacts are changed, added, or removed. But those signals sends lists of entire EContacts, containing the entire contact data, though we often have all the data already and just want to know which contacts have changed.

[http://developer.gnome.org/libebook/stable/EBook.html#e-book-get-book-view e_book_get_book_view()] takes an [http://developer.gnome.org/libebook/stable/libebook-e-book-query.html#EBookQuery EBookQuery] which could be extended to take a new flag to make the EBookView send only the UID in the signals' EContacts. For instance, "X-EDS-QUERY-FLAG-NOTIFY-UID-ONLY". The UID is already stored separately for use as the primary key of the local-storage backend's BDB database, so no VCard parsing should be necessary to provide it.

This would allow easier integration with the QContact API, greatly improving performance.

Note that users of this feature should probably use just the base EVCard API on the provided EContact objects, to avoid extra processing and caching in the EContact code.

Note: We previously considered implementing e_book_get_book_view()'s requested fields parameter, but that changes the whole view, though we only really want to change what is sent for changed/new contacts.

Note: Chris discovered a dedicated e_book_get_contact_uids() API in the Fremantle EDS fork. The new API call was accepted upstream (in master) and [https://bugzilla.gnome.org/show_bug.cgi?id=651446#c11 backported as a patch] (used in the Meego package) for the gnome-2-32 branch, but this is not a full solution.

[https://bugzilla.gnome.org/show_bug.cgi?id=652172 Upstream evolution-data-server bug]

== Optional Parsing of ebook Data in Client ==

The EDS client-side APIs currently parse and (temporarily) store the parsed details of VCards. However, this is often unnecessary because APIs such as e_book_get_book_view() cause it to return the whole VCard anyway. There will be some performance gain from delaying VCard parsing until it is really needed by calls to functions such as e_contact_new_from_vcard().

libebook provides different functionality:
# exchange data as vCard with EDS daemon
# parse items and wrap them in a C API

QtContacts replaces the second part. An obvious performance improvement is to only do the second part if the additional C APIs are really called. The following calls then transfer data without any additional parsing in libebook:
* e_book_get_contact()/asynchronous read/view + e_vcard_to_string()
* e_contact_new_from_vcard() + e_book_add_contact()/e_book_commit_contact()/asynchronous storing

Note that libecal (for iCalendar) can already do this, though libebook can't yet.

TODO:
# add delayed parsing to libebook, triggered by API call which needs parsed data

PROBLEM:
# If delayed parsing fails, how can error be reported?
# Should delayed parsing be done always (e_vcard_to_string())) or only if requested? How would that be configured by a libebook user?

An [http://build.meego.com/package/view_file?file=EBookView-Implement-delayed-vcard-parsing.patch&package=evolution-data-server&project=eds&srcmd5=977d2b885bed818cb616599749407e79 e_book_view_set_parse_vcards() patch], already used in the Meego package, partly implements this already. This patch was taken from an old fremantle EDS patch that was never actually used, and is generally considered to be a bad hack that should not be accepted upstream.

== Asynchronous API for storing items in libecal ==

libecal should allow the caller to request storing of data without blocking on processing of the data. It should notify the caller when the data has been stored, or when processing has failed.

These functions may require async versions:
* icalcomponent_new_from_string()
* e_cal_modify_object[_with_mod]
* e_cal_create_object()

Making libical asynchronous is almost impossible without going multithreaded. Let's focus on e_cal D-Bus communication first.

Rob started some work on this in the rbradford-wip-ecal-async-api branch. Perhaps you can reuse some of that work. It is not functional, but might be a source of inspiration or code fragments.

== More Efficient Access to Meta Data for Change Tracking ==

This was discussed a while ago: [[http://www.mail-archive.com/evolution-hackers@gnome.org/msg03029.html
[Evolution-hackers] concurrent modifications of items in GUI and EDS database]]

Data synchronization only needs to know which items exist (list of local IDs) and which revision of each item is currently stored. Revisions can be identified with a string which changes for each modification. Restoring an older revision should (but doesn't have to) restore the revision string.

For calendar, UID+RECURRENCE-ID and LAST-MODIFIED can be used. For contacts, UID + REV.

By comparing a stored list against the current one, changes can be detected without relying on journals in the daemon or comparison of time stamps. Journals are hard to maintain (how many of them, how long?). Comparison of time stamps has problems when changes are allowed while a sync runs and assumes that system time is linear (which is not always true).

As an additional use case, local ID + revision string could be used to prevent overwriting stale data in the server, as in HTTP PUT eTag checks.

TODO:

The Maemo Fremantle port of EDS, which is of course already open source, apparently has some solution for this, according to Mathias Hasselman, so that should be investigated first.

However, this should probably be made possible by:

# Extending the EDS query language with a flag that limits the delivered data to just:
## UID and REV for libebook (REV is in VCard 3.0)
## UID+RECURRENCE-ID+LAST-MODIFIED, for libecal
# Implementing these queries for the local contact and calendar backends. The query would be rejected, with an error, for other backends.

== Atomic Updates ==

Later, implement atomic changes as discussed in the mail thread: [[http://www.mail-archive.com/evolution-hackers@gnome.org/msg03029.html

== Quick check for "data changed" ==

The libebook and libical APIs should offer a way to discover if any data whatsoever has changed. This would, for instance, help SyncEvolution to avoid unnecessary detailed checking. This could just check the modification dates of local storage files, such as the BDB .db file, or .ics file, as long as we do not use those files to store permanent caches of parsed data. This would be much quicker than listing items for the whole database and comparing their revisions strings, as is currently necessary.

This would require small API additions, to check that the opaque revision strings are unchanged, such as

gchar* e_book_get_revision(EBook *book)
gboolean e_book_check_revision(EBook *book , const gchar* revision)

and

gchar* e_cal_get_revision(ECal *ecal)
gboolean e_cal_check_revision(ECal ecal, const gchar revision)

The semantic would be this:
* If the revision string is empty or different, then the content of the database '''might''' have changed. A more detailed check is necessary to determine if it has really changed. This will be the case for non-local storage.
* If it is non-empty and the same as before, the nothing has changed. No further check will be necessary.

== Contacts: Store PHOTO Data as Plain Files ==

Photos should be transferred over D-Bus via a filepath to a local file rather than transferring the actual photo data over D-Bus. The VCard format already allows this option (See the PHOTO property). We must be careful to keep these local files in sync and to remove them when contacts are removed, and to restrict access to the files from other processes.

There should be utility code available to libebook users to convert between the different representations.

Goals:
# Apps should be able to create photo files without ever passing the data to libebook.
# External PHOTO file and corresponding contact must be kept in sync:
** If a contact is deleted, the photo also needs to be removed.
** If a photo file is created and then storing the contact fails, who removes the file?
# Access to photo data must be limited to processes which have the right permissions.

Proposal:
* For each ~/.local/share/evolution/foo calendar, store photos in <photo path>=~/.local/share/evolution/foo/photos, with same permissions.
* New photo files are created with a [http://en.wikipedia.org/wiki/Universally_unique_identifier UUID] as base name and suffix according to the file content.
* If a contact is deleted and has a PHOTO;VALUE=uri:file://<photo path>/*, delete that file.
* For apps which want to create the photo file directly:
** Apps get a file descriptor and the path from a new libebook API.
** They own the file until the EDS server acknowledges the successful storing of a contact with a matching PHOTO;VALUE=uri:file property. If that fails, the app must remove the file.
* Add new API calls to libebook which split out resp. include data inline.
** Must track ownership of file after splitting it out and before storing - might never get stored!

Open:
* Should there be automatic garbage collection of stale photos in case something went wrong and a photo wasn't properly deleted?
* Trust TYPE=JPEG/PNG/... or look into file content to determine type?

== Future: Content Protection ==

Only authorized applications (= processes) with the necessary permissions should be able to read/write data. Exact permissions and mechanisms for verifying them to be defined as part of the (revised?) MeeGo architecture.

This can be added as follows:
# lock down file permissions
# add permission checks to EDS D-Bus servers

Currently not planned. Depends on the future MeeGo security infrastructure.

Architecture/planning/evolution-data-server/eds-improvements

2011-06-09T09:48:10Z

Murrayc: /* Optional Parsing of ebook Data in Client */

This page describes improvements that are desirable or even essential for good performance of Evolution Data Server (EDS) as PIM storage in MeeGo.

All of these changes should be developed for the evolution-data-server gnome-2-32 branch, as used by MeeGo 1.2, then ported to master for review by upstream maintainers. Most changes will not be accepted in upstream's gnome-2-32 branch, because is is stable, but Meego will apply them as patches to its package.

== Change Tracking ==

EDS is based on the model/view/controller principle: an application opens a database (calendar or addressbook) and requests a view with server-side filtering, then the application is sent:
# all data covered by the view
# the "all data sent" signal
# changes (add/update/remove) while the view exists

Applications modify data with asynchronous requests (controller) and keep showing the old data until the change notification comes in. The exact changes made to the data are determined by the server (for example when meeting invitations are sent).

QtContacts and KCalCore do not have such a strict model. With QtContacts the app simply loads and stores contacts and is responsible for maintaining its own view. Changes are sent for changes made after the database was opened. KCalCore is based on the concept of loading a calendar (partially) into the client, modifying it there, then writing it back. Changes are also sent while the storage is open.

TODO:
* https://bugs.meego.com/show_bug.cgi?id=18760 - "QtContacts-EDS + EDS: more flexible EBookView (UID only, only future changes)"

=== Suppress Unnecessary Data Transfer ===

EDS currently sends all data when a view is requested, which is unnecessary when the client is only interested in change notifications. An extension to the EDS query language could optionally suppresses that sending of existing data when the view is opened, to be implemented only for the local contacts and calendar backends. This would allow easier integration with the QContact API.

The Fremantle EDS fork also had a “freezable” (static) capability for libebook, which should probably be investigated. See, for instance: http://maemo.org/api_refs/5.0/5.0-final/libebook/EBookView.html#e-book-view-set-freezable

One possibility for adding the flag is to (ab)use the existing query language with a match for special fields:
* in the client: query = e_book_query_orv(e_book_query_field_exists("X-EDS-QUERY-FLAG-SKIP-DATA-DUMP"), <real query>, NULL);
* in the server: "if (query is OR-Query and first clause is exists("X-EDS-QUERY-FLAG-SKIP-DATA-DUMP")) { skip_data_dump = TRUE; query = second clause; }

[https://bugzilla.gnome.org/show_bug.cgi?id=652171 Upstream evolution-data-server bug]

=== Nofify with UIDs Only ===

EBookView emits signals when contacts are changed, added, or removed. But those signals sends lists of entire EContacts, containing the entire contact data, though we often have all the data already and just want to know which contacts have changed.

[http://developer.gnome.org/libebook/stable/EBook.html#e-book-get-book-view e_book_get_book_view()] takes an [http://developer.gnome.org/libebook/stable/libebook-e-book-query.html#EBookQuery EBookQuery] which could be extended to take a new flag to make the EBookView send only the UID in the signals' EContacts. For instance, "X-EDS-QUERY-FLAG-NOTIFY-UID-ONLY". The UID is already stored separately for use as the primary key of the local-storage backend's BDB database, so no VCard parsing should be necessary to provide it.

This would allow easier integration with the QContact API, greatly improving performance.

Note that users of this feature should probably use just the base EVCard API on the provided EContact objects, to avoid extra processing and caching in the EContact code.

Note: We previously considered implementing e_book_get_book_view()'s requested fields parameter, but that changes the whole view, though we only really want to change what is sent for changed/new contacts.

Note: Chris discovered a dedicated e_book_get_contact_uids() API in the Fremantle EDS fork. The new API call was accepted upstream (in master) and [https://bugzilla.gnome.org/show_bug.cgi?id=651446#c11 backported as a patch] (used in the Meego package) for the gnome-2-32 branch, but this is not a full solution.

[https://bugzilla.gnome.org/show_bug.cgi?id=652172 Upstream evolution-data-server bug]

== Optional Parsing of ebook Data in Client ==

The EDS client-side APIs currently parse and (temporarily) store the parsed details of VCards. However, this is often unnecessary because APIs such as e_book_get_book_view() cause it to return the whole VCard anyway. There will be some performance gain from delaying VCard parsing until it is really needed by calls to functions such as e_contact_new_from_vcard().

libebook provides different functionality:
# exchange data as vCard with EDS daemon
# parse items and wrap them in a C API

QtContacts replaces the second part. An obvious performance improvement is to only do the second part if the additional C APIs are really called. The following calls then transfer data without any additional parsing in libebook:
* e_book_get_contact()/asynchronous read/view + e_vcard_to_string()
* e_contact_new_from_vcard() + e_book_add_contact()/e_book_commit_contact()/asynchronous storing

Note that libecal (for iCalendar) can already do this, though libebook can't yet.

TODO:
# add delayed parsing to libebook, triggered by API call which needs parsed data

PROBLEM:
# If delayed parsing fails, how can error be reported?
# Should delayed parsing be done always (e_vcard_to_string())) or only if requested? How would that be configured by a libebook user?

An existing (and in use) [http://build.meego.com/package/view_file?file=EBookView-Implement-delayed-vcard-parsing.patch&package=evolution-data-server&project=eds&srcmd5=977d2b885bed818cb616599749407e79 e_book_view_set_parse_vcards() patch] partly implements this already. This patch was taken from an old fremantle EDS patch that was never actually used, and is generally considered to be a bad hack that should not be accepted upstream.

== Asynchronous API for storing items in libecal ==

libecal should allow the caller to request storing of data without blocking on processing of the data. It should notify the caller when the data has been stored, or when processing has failed.

These functions may require async versions:
* icalcomponent_new_from_string()
* e_cal_modify_object[_with_mod]
* e_cal_create_object()

Making libical asynchronous is almost impossible without going multithreaded. Let's focus on e_cal D-Bus communication first.

Rob started some work on this in the rbradford-wip-ecal-async-api branch. Perhaps you can reuse some of that work. It is not functional, but might be a source of inspiration or code fragments.

== More Efficient Access to Meta Data for Change Tracking ==

This was discussed a while ago: [[http://www.mail-archive.com/evolution-hackers@gnome.org/msg03029.html
[Evolution-hackers] concurrent modifications of items in GUI and EDS database]]

Data synchronization only needs to know which items exist (list of local IDs) and which revision of each item is currently stored. Revisions can be identified with a string which changes for each modification. Restoring an older revision should (but doesn't have to) restore the revision string.

For calendar, UID+RECURRENCE-ID and LAST-MODIFIED can be used. For contacts, UID + REV.

By comparing a stored list against the current one, changes can be detected without relying on journals in the daemon or comparison of time stamps. Journals are hard to maintain (how many of them, how long?). Comparison of time stamps has problems when changes are allowed while a sync runs and assumes that system time is linear (which is not always true).

As an additional use case, local ID + revision string could be used to prevent overwriting stale data in the server, as in HTTP PUT eTag checks.

TODO:

The Maemo Fremantle port of EDS, which is of course already open source, apparently has some solution for this, according to Mathias Hasselman, so that should be investigated first.

However, this should probably be made possible by:

# Extending the EDS query language with a flag that limits the delivered data to just:
## UID and REV for libebook (REV is in VCard 3.0)
## UID+RECURRENCE-ID+LAST-MODIFIED, for libecal
# Implementing these queries for the local contact and calendar backends. The query would be rejected, with an error, for other backends.

== Atomic Updates ==

Later, implement atomic changes as discussed in the mail thread: [[http://www.mail-archive.com/evolution-hackers@gnome.org/msg03029.html

== Quick check for "data changed" ==

The libebook and libical APIs should offer a way to discover if any data whatsoever has changed. This would, for instance, help SyncEvolution to avoid unnecessary detailed checking. This could just check the modification dates of local storage files, such as the BDB .db file, or .ics file, as long as we do not use those files to store permanent caches of parsed data. This would be much quicker than listing items for the whole database and comparing their revisions strings, as is currently necessary.

This would require small API additions, to check that the opaque revision strings are unchanged, such as

gchar* e_book_get_revision(EBook *book)
gboolean e_book_check_revision(EBook *book , const gchar* revision)

and

gchar* e_cal_get_revision(ECal *ecal)
gboolean e_cal_check_revision(ECal ecal, const gchar revision)

The semantic would be this:
* If the revision string is empty or different, then the content of the database '''might''' have changed. A more detailed check is necessary to determine if it has really changed. This will be the case for non-local storage.
* If it is non-empty and the same as before, the nothing has changed. No further check will be necessary.

== Contacts: Store PHOTO Data as Plain Files ==

Photos should be transferred over D-Bus via a filepath to a local file rather than transferring the actual photo data over D-Bus. The VCard format already allows this option (See the PHOTO property). We must be careful to keep these local files in sync and to remove them when contacts are removed, and to restrict access to the files from other processes.

There should be utility code available to libebook users to convert between the different representations.

Goals:
# Apps should be able to create photo files without ever passing the data to libebook.
# External PHOTO file and corresponding contact must be kept in sync:
** If a contact is deleted, the photo also needs to be removed.
** If a photo file is created and then storing the contact fails, who removes the file?
# Access to photo data must be limited to processes which have the right permissions.

Proposal:
* For each ~/.local/share/evolution/foo calendar, store photos in <photo path>=~/.local/share/evolution/foo/photos, with same permissions.
* New photo files are created with a [http://en.wikipedia.org/wiki/Universally_unique_identifier UUID] as base name and suffix according to the file content.
* If a contact is deleted and has a PHOTO;VALUE=uri:file://<photo path>/*, delete that file.
* For apps which want to create the photo file directly:
** Apps get a file descriptor and the path from a new libebook API.
** They own the file until the EDS server acknowledges the successful storing of a contact with a matching PHOTO;VALUE=uri:file property. If that fails, the app must remove the file.
* Add new API calls to libebook which split out resp. include data inline.
** Must track ownership of file after splitting it out and before storing - might never get stored!

Open:
* Should there be automatic garbage collection of stale photos in case something went wrong and a photo wasn't properly deleted?
* Trust TYPE=JPEG/PNG/... or look into file content to determine type?

== Future: Content Protection ==

Only authorized applications (= processes) with the necessary permissions should be able to read/write data. Exact permissions and mechanisms for verifying them to be defined as part of the (revised?) MeeGo architecture.

This can be added as follows:
# lock down file permissions
# add permission checks to EDS D-Bus servers

Currently not planned. Depends on the future MeeGo security infrastructure.

Architecture/planning/evolution-data-server/eds-improvements

2011-06-09T09:46:08Z

Murrayc: /* Nofify with UIDs Only */

This page describes improvements that are desirable or even essential for good performance of Evolution Data Server (EDS) as PIM storage in MeeGo.

All of these changes should be developed for the evolution-data-server gnome-2-32 branch, as used by MeeGo 1.2, then ported to master for review by upstream maintainers. Most changes will not be accepted in upstream's gnome-2-32 branch, because is is stable, but Meego will apply them as patches to its package.

== Change Tracking ==

EDS is based on the model/view/controller principle: an application opens a database (calendar or addressbook) and requests a view with server-side filtering, then the application is sent:
# all data covered by the view
# the "all data sent" signal
# changes (add/update/remove) while the view exists

Applications modify data with asynchronous requests (controller) and keep showing the old data until the change notification comes in. The exact changes made to the data are determined by the server (for example when meeting invitations are sent).

QtContacts and KCalCore do not have such a strict model. With QtContacts the app simply loads and stores contacts and is responsible for maintaining its own view. Changes are sent for changes made after the database was opened. KCalCore is based on the concept of loading a calendar (partially) into the client, modifying it there, then writing it back. Changes are also sent while the storage is open.

TODO:
* https://bugs.meego.com/show_bug.cgi?id=18760 - "QtContacts-EDS + EDS: more flexible EBookView (UID only, only future changes)"

=== Suppress Unnecessary Data Transfer ===

EDS currently sends all data when a view is requested, which is unnecessary when the client is only interested in change notifications. An extension to the EDS query language could optionally suppresses that sending of existing data when the view is opened, to be implemented only for the local contacts and calendar backends. This would allow easier integration with the QContact API.

The Fremantle EDS fork also had a “freezable” (static) capability for libebook, which should probably be investigated. See, for instance: http://maemo.org/api_refs/5.0/5.0-final/libebook/EBookView.html#e-book-view-set-freezable

One possibility for adding the flag is to (ab)use the existing query language with a match for special fields:
* in the client: query = e_book_query_orv(e_book_query_field_exists("X-EDS-QUERY-FLAG-SKIP-DATA-DUMP"), <real query>, NULL);
* in the server: "if (query is OR-Query and first clause is exists("X-EDS-QUERY-FLAG-SKIP-DATA-DUMP")) { skip_data_dump = TRUE; query = second clause; }

[https://bugzilla.gnome.org/show_bug.cgi?id=652171 Upstream evolution-data-server bug]

=== Nofify with UIDs Only ===

EBookView emits signals when contacts are changed, added, or removed. But those signals sends lists of entire EContacts, containing the entire contact data, though we often have all the data already and just want to know which contacts have changed.

[http://developer.gnome.org/libebook/stable/EBook.html#e-book-get-book-view e_book_get_book_view()] takes an [http://developer.gnome.org/libebook/stable/libebook-e-book-query.html#EBookQuery EBookQuery] which could be extended to take a new flag to make the EBookView send only the UID in the signals' EContacts. For instance, "X-EDS-QUERY-FLAG-NOTIFY-UID-ONLY". The UID is already stored separately for use as the primary key of the local-storage backend's BDB database, so no VCard parsing should be necessary to provide it.

This would allow easier integration with the QContact API, greatly improving performance.

Note that users of this feature should probably use just the base EVCard API on the provided EContact objects, to avoid extra processing and caching in the EContact code.

Note: We previously considered implementing e_book_get_book_view()'s requested fields parameter, but that changes the whole view, though we only really want to change what is sent for changed/new contacts.

Note: Chris discovered a dedicated e_book_get_contact_uids() API in the Fremantle EDS fork. The new API call was accepted upstream (in master) and [https://bugzilla.gnome.org/show_bug.cgi?id=651446#c11 backported as a patch] (used in the Meego package) for the gnome-2-32 branch, but this is not a full solution.

[https://bugzilla.gnome.org/show_bug.cgi?id=652172 Upstream evolution-data-server bug]

== Optional Parsing of ebook Data in Client ==

The EDS client-side APIs currently parse and (temporarily) store the parsed details of VCards. However, this is often unnecessary because APIs such as e_book_get_book_view() cause it to return the whole VCard anyway. There will be some performance gain from delaying VCard parsing until it is really needed by calls to functions such as e_contact_new_from_vcard().

libebook provides different functionality:
# exchange data as vCard with EDS daemon
# parse items and wrap them in a C API

QtContacts replaces the second part. An obvious performance improvement is to only do the second part if the additional C APIs are really called. The following calls then transfer data without any additional parsing in libebook:
* e_book_get_contact()/asynchronous read/view + e_vcard_to_string()
* e_contact_new_from_vcard() + e_book_add_contact()/e_book_commit_contact()/asynchronous storing

Note that libecal (for iCalendar) can already do this, though libebook can't yet.

TODO:
# add delayed parsing to libebook, triggered by API call which needs parsed data

PROBLEM:
# If delayed parsing fails, how can error be reported?
# Should delayed parsing be done always (e_vcard_to_string())) or only if requested? How would that be configured by a libebook user?

An existing (and in use) [http://build.meego.com/package/view_file?file=EBookView-Implement-delayed-vcard-parsing.patch&package=evolution-data-server&project=eds&srcmd5=977d2b885bed818cb616599749407e79 patch] might partly implement this already.

== Asynchronous API for storing items in libecal ==

libecal should allow the caller to request storing of data without blocking on processing of the data. It should notify the caller when the data has been stored, or when processing has failed.

These functions may require async versions:
* icalcomponent_new_from_string()
* e_cal_modify_object[_with_mod]
* e_cal_create_object()

Making libical asynchronous is almost impossible without going multithreaded. Let's focus on e_cal D-Bus communication first.

Rob started some work on this in the rbradford-wip-ecal-async-api branch. Perhaps you can reuse some of that work. It is not functional, but might be a source of inspiration or code fragments.

== More Efficient Access to Meta Data for Change Tracking ==

This was discussed a while ago: [[http://www.mail-archive.com/evolution-hackers@gnome.org/msg03029.html
[Evolution-hackers] concurrent modifications of items in GUI and EDS database]]

Data synchronization only needs to know which items exist (list of local IDs) and which revision of each item is currently stored. Revisions can be identified with a string which changes for each modification. Restoring an older revision should (but doesn't have to) restore the revision string.

For calendar, UID+RECURRENCE-ID and LAST-MODIFIED can be used. For contacts, UID + REV.

By comparing a stored list against the current one, changes can be detected without relying on journals in the daemon or comparison of time stamps. Journals are hard to maintain (how many of them, how long?). Comparison of time stamps has problems when changes are allowed while a sync runs and assumes that system time is linear (which is not always true).

As an additional use case, local ID + revision string could be used to prevent overwriting stale data in the server, as in HTTP PUT eTag checks.

TODO:

The Maemo Fremantle port of EDS, which is of course already open source, apparently has some solution for this, according to Mathias Hasselman, so that should be investigated first.

However, this should probably be made possible by:

# Extending the EDS query language with a flag that limits the delivered data to just:
## UID and REV for libebook (REV is in VCard 3.0)
## UID+RECURRENCE-ID+LAST-MODIFIED, for libecal
# Implementing these queries for the local contact and calendar backends. The query would be rejected, with an error, for other backends.

== Atomic Updates ==

Later, implement atomic changes as discussed in the mail thread: [[http://www.mail-archive.com/evolution-hackers@gnome.org/msg03029.html

== Quick check for "data changed" ==

The libebook and libical APIs should offer a way to discover if any data whatsoever has changed. This would, for instance, help SyncEvolution to avoid unnecessary detailed checking. This could just check the modification dates of local storage files, such as the BDB .db file, or .ics file, as long as we do not use those files to store permanent caches of parsed data. This would be much quicker than listing items for the whole database and comparing their revisions strings, as is currently necessary.

This would require small API additions, to check that the opaque revision strings are unchanged, such as

gchar* e_book_get_revision(EBook *book)
gboolean e_book_check_revision(EBook *book , const gchar* revision)

and

gchar* e_cal_get_revision(ECal *ecal)
gboolean e_cal_check_revision(ECal ecal, const gchar revision)

The semantic would be this:
* If the revision string is empty or different, then the content of the database '''might''' have changed. A more detailed check is necessary to determine if it has really changed. This will be the case for non-local storage.
* If it is non-empty and the same as before, the nothing has changed. No further check will be necessary.

== Contacts: Store PHOTO Data as Plain Files ==

Photos should be transferred over D-Bus via a filepath to a local file rather than transferring the actual photo data over D-Bus. The VCard format already allows this option (See the PHOTO property). We must be careful to keep these local files in sync and to remove them when contacts are removed, and to restrict access to the files from other processes.

There should be utility code available to libebook users to convert between the different representations.

Goals:
# Apps should be able to create photo files without ever passing the data to libebook.
# External PHOTO file and corresponding contact must be kept in sync:
** If a contact is deleted, the photo also needs to be removed.
** If a photo file is created and then storing the contact fails, who removes the file?
# Access to photo data must be limited to processes which have the right permissions.

Proposal:
* For each ~/.local/share/evolution/foo calendar, store photos in <photo path>=~/.local/share/evolution/foo/photos, with same permissions.
* New photo files are created with a [http://en.wikipedia.org/wiki/Universally_unique_identifier UUID] as base name and suffix according to the file content.
* If a contact is deleted and has a PHOTO;VALUE=uri:file://<photo path>/*, delete that file.
* For apps which want to create the photo file directly:
** Apps get a file descriptor and the path from a new libebook API.
** They own the file until the EDS server acknowledges the successful storing of a contact with a matching PHOTO;VALUE=uri:file property. If that fails, the app must remove the file.
* Add new API calls to libebook which split out resp. include data inline.
** Must track ownership of file after splitting it out and before storing - might never get stored!

Open:
* Should there be automatic garbage collection of stale photos in case something went wrong and a photo wasn't properly deleted?
* Trust TYPE=JPEG/PNG/... or look into file content to determine type?

== Future: Content Protection ==

Only authorized applications (= processes) with the necessary permissions should be able to read/write data. Exact permissions and mechanisms for verifying them to be defined as part of the (revised?) MeeGo architecture.

This can be added as follows:
# lock down file permissions
# add permission checks to EDS D-Bus servers

Currently not planned. Depends on the future MeeGo security infrastructure.

Architecture/planning/evolution-data-server/eds-improvements

2011-06-09T09:42:57Z

Murrayc: /* Suppress Unnecessary Data Transfer */

This page describes improvements that are desirable or even essential for good performance of Evolution Data Server (EDS) as PIM storage in MeeGo.

All of these changes should be developed for the evolution-data-server gnome-2-32 branch, as used by MeeGo 1.2, then ported to master for review by upstream maintainers. Most changes will not be accepted in upstream's gnome-2-32 branch, because is is stable, but Meego will apply them as patches to its package.

== Change Tracking ==

EDS is based on the model/view/controller principle: an application opens a database (calendar or addressbook) and requests a view with server-side filtering, then the application is sent:
# all data covered by the view
# the "all data sent" signal
# changes (add/update/remove) while the view exists

Applications modify data with asynchronous requests (controller) and keep showing the old data until the change notification comes in. The exact changes made to the data are determined by the server (for example when meeting invitations are sent).

QtContacts and KCalCore do not have such a strict model. With QtContacts the app simply loads and stores contacts and is responsible for maintaining its own view. Changes are sent for changes made after the database was opened. KCalCore is based on the concept of loading a calendar (partially) into the client, modifying it there, then writing it back. Changes are also sent while the storage is open.

TODO:
* https://bugs.meego.com/show_bug.cgi?id=18760 - "QtContacts-EDS + EDS: more flexible EBookView (UID only, only future changes)"

=== Suppress Unnecessary Data Transfer ===

EDS currently sends all data when a view is requested, which is unnecessary when the client is only interested in change notifications. An extension to the EDS query language could optionally suppresses that sending of existing data when the view is opened, to be implemented only for the local contacts and calendar backends. This would allow easier integration with the QContact API.

The Fremantle EDS fork also had a “freezable” (static) capability for libebook, which should probably be investigated. See, for instance: http://maemo.org/api_refs/5.0/5.0-final/libebook/EBookView.html#e-book-view-set-freezable

One possibility for adding the flag is to (ab)use the existing query language with a match for special fields:
* in the client: query = e_book_query_orv(e_book_query_field_exists("X-EDS-QUERY-FLAG-SKIP-DATA-DUMP"), <real query>, NULL);
* in the server: "if (query is OR-Query and first clause is exists("X-EDS-QUERY-FLAG-SKIP-DATA-DUMP")) { skip_data_dump = TRUE; query = second clause; }

[https://bugzilla.gnome.org/show_bug.cgi?id=652171 Upstream evolution-data-server bug]

=== Nofify with UIDs Only ===

EBookView emits signals when contacts are changed, added, or removed. But those signals sends lists of entire EContacts, containing the entire contact data, though we often have all the data already and just want to know which contacts have changed. This would allow easier integration with the QContact API, greatly improving performance.

[http://developer.gnome.org/libebook/stable/EBook.html#e-book-get-book-view e_book_get_book_view()] takes an [http://developer.gnome.org/libebook/stable/libebook-e-book-query.html#EBookQuery EBookQuery] which could be extended to take a new flag to make the EBookView send only the UID in the signals' EContacts. For instance, "X-EDS-QUERY-FLAG-NOTIFY-UID-ONLY". The UID is already stored separately for use as the primary key of the local-storage backend's BDB database, so no VCard parsing should be necessary to provide it.

Note that users of this feature should probably use just the base EVCard API on the provided EContact objects, to avoid extra processing and caching in the EContact code.

Note: We previously considered implementing e_book_get_book_view()'s requested fields parameter, but that changes the whole view, though we only really want to change what is sent for changed/new contacts.

Note: Chris discovered a dedicated e_book_get_contact_uids() API in the Fremantle EDS fork. The new API call was accepted upstream (in master) and [https://bugzilla.gnome.org/show_bug.cgi?id=651446#c11 backported as a patch] (used in the Meego package) for the gnome-2-32 branch, but this is not a full solution.

== Optional Parsing of ebook Data in Client ==

The EDS client-side APIs currently parse and (temporarily) store the parsed details of VCards. However, this is often unnecessary because APIs such as e_book_get_book_view() cause it to return the whole VCard anyway. There will be some performance gain from delaying VCard parsing until it is really needed by calls to functions such as e_contact_new_from_vcard().

libebook provides different functionality:
# exchange data as vCard with EDS daemon
# parse items and wrap them in a C API

QtContacts replaces the second part. An obvious performance improvement is to only do the second part if the additional C APIs are really called. The following calls then transfer data without any additional parsing in libebook:
* e_book_get_contact()/asynchronous read/view + e_vcard_to_string()
* e_contact_new_from_vcard() + e_book_add_contact()/e_book_commit_contact()/asynchronous storing

Note that libecal (for iCalendar) can already do this, though libebook can't yet.

TODO:
# add delayed parsing to libebook, triggered by API call which needs parsed data

PROBLEM:
# If delayed parsing fails, how can error be reported?
# Should delayed parsing be done always (e_vcard_to_string())) or only if requested? How would that be configured by a libebook user?

An existing (and in use) [http://build.meego.com/package/view_file?file=EBookView-Implement-delayed-vcard-parsing.patch&package=evolution-data-server&project=eds&srcmd5=977d2b885bed818cb616599749407e79 patch] might partly implement this already.

== Asynchronous API for storing items in libecal ==

libecal should allow the caller to request storing of data without blocking on processing of the data. It should notify the caller when the data has been stored, or when processing has failed.

These functions may require async versions:
* icalcomponent_new_from_string()
* e_cal_modify_object[_with_mod]
* e_cal_create_object()

Making libical asynchronous is almost impossible without going multithreaded. Let's focus on e_cal D-Bus communication first.

Rob started some work on this in the rbradford-wip-ecal-async-api branch. Perhaps you can reuse some of that work. It is not functional, but might be a source of inspiration or code fragments.

== More Efficient Access to Meta Data for Change Tracking ==

This was discussed a while ago: [[http://www.mail-archive.com/evolution-hackers@gnome.org/msg03029.html
[Evolution-hackers] concurrent modifications of items in GUI and EDS database]]

Data synchronization only needs to know which items exist (list of local IDs) and which revision of each item is currently stored. Revisions can be identified with a string which changes for each modification. Restoring an older revision should (but doesn't have to) restore the revision string.

For calendar, UID+RECURRENCE-ID and LAST-MODIFIED can be used. For contacts, UID + REV.

By comparing a stored list against the current one, changes can be detected without relying on journals in the daemon or comparison of time stamps. Journals are hard to maintain (how many of them, how long?). Comparison of time stamps has problems when changes are allowed while a sync runs and assumes that system time is linear (which is not always true).

As an additional use case, local ID + revision string could be used to prevent overwriting stale data in the server, as in HTTP PUT eTag checks.

TODO:

The Maemo Fremantle port of EDS, which is of course already open source, apparently has some solution for this, according to Mathias Hasselman, so that should be investigated first.

However, this should probably be made possible by:

# Extending the EDS query language with a flag that limits the delivered data to just:
## UID and REV for libebook (REV is in VCard 3.0)
## UID+RECURRENCE-ID+LAST-MODIFIED, for libecal
# Implementing these queries for the local contact and calendar backends. The query would be rejected, with an error, for other backends.

== Atomic Updates ==

Later, implement atomic changes as discussed in the mail thread: [[http://www.mail-archive.com/evolution-hackers@gnome.org/msg03029.html

== Quick check for "data changed" ==

The libebook and libical APIs should offer a way to discover if any data whatsoever has changed. This would, for instance, help SyncEvolution to avoid unnecessary detailed checking. This could just check the modification dates of local storage files, such as the BDB .db file, or .ics file, as long as we do not use those files to store permanent caches of parsed data. This would be much quicker than listing items for the whole database and comparing their revisions strings, as is currently necessary.

This would require small API additions, to check that the opaque revision strings are unchanged, such as

gchar* e_book_get_revision(EBook *book)
gboolean e_book_check_revision(EBook *book , const gchar* revision)

and

gchar* e_cal_get_revision(ECal *ecal)
gboolean e_cal_check_revision(ECal ecal, const gchar revision)

The semantic would be this:
* If the revision string is empty or different, then the content of the database '''might''' have changed. A more detailed check is necessary to determine if it has really changed. This will be the case for non-local storage.
* If it is non-empty and the same as before, the nothing has changed. No further check will be necessary.

== Contacts: Store PHOTO Data as Plain Files ==

Photos should be transferred over D-Bus via a filepath to a local file rather than transferring the actual photo data over D-Bus. The VCard format already allows this option (See the PHOTO property). We must be careful to keep these local files in sync and to remove them when contacts are removed, and to restrict access to the files from other processes.

There should be utility code available to libebook users to convert between the different representations.

Goals:
# Apps should be able to create photo files without ever passing the data to libebook.
# External PHOTO file and corresponding contact must be kept in sync:
** If a contact is deleted, the photo also needs to be removed.
** If a photo file is created and then storing the contact fails, who removes the file?
# Access to photo data must be limited to processes which have the right permissions.

Proposal:
* For each ~/.local/share/evolution/foo calendar, store photos in <photo path>=~/.local/share/evolution/foo/photos, with same permissions.
* New photo files are created with a [http://en.wikipedia.org/wiki/Universally_unique_identifier UUID] as base name and suffix according to the file content.
* If a contact is deleted and has a PHOTO;VALUE=uri:file://<photo path>/*, delete that file.
* For apps which want to create the photo file directly:
** Apps get a file descriptor and the path from a new libebook API.
** They own the file until the EDS server acknowledges the successful storing of a contact with a matching PHOTO;VALUE=uri:file property. If that fails, the app must remove the file.
* Add new API calls to libebook which split out resp. include data inline.
** Must track ownership of file after splitting it out and before storing - might never get stored!

Open:
* Should there be automatic garbage collection of stale photos in case something went wrong and a photo wasn't properly deleted?
* Trust TYPE=JPEG/PNG/... or look into file content to determine type?

== Future: Content Protection ==

Only authorized applications (= processes) with the necessary permissions should be able to read/write data. Exact permissions and mechanisms for verifying them to be defined as part of the (revised?) MeeGo architecture.

This can be added as follows:
# lock down file permissions
# add permission checks to EDS D-Bus servers

Currently not planned. Depends on the future MeeGo security infrastructure.

Architecture/planning/evolution-data-server/eds-improvements

2011-06-09T09:29:50Z

Murrayc: Return UIDs Only: Let's change the notification signals instead.

This page describes improvements that are desirable or even essential for good performance of Evolution Data Server (EDS) as PIM storage in MeeGo.

All of these changes should be developed for the evolution-data-server gnome-2-32 branch, as used by MeeGo 1.2, then ported to master for review by upstream maintainers. Most changes will not be accepted in upstream's gnome-2-32 branch, because is is stable, but Meego will apply them as patches to its package.

== Change Tracking ==

EDS is based on the model/view/controller principle: an application opens a database (calendar or addressbook) and requests a view with server-side filtering, then the application is sent:
# all data covered by the view
# the "all data sent" signal
# changes (add/update/remove) while the view exists

Applications modify data with asynchronous requests (controller) and keep showing the old data until the change notification comes in. The exact changes made to the data are determined by the server (for example when meeting invitations are sent).

QtContacts and KCalCore do not have such a strict model. With QtContacts the app simply loads and stores contacts and is responsible for maintaining its own view. Changes are sent for changes made after the database was opened. KCalCore is based on the concept of loading a calendar (partially) into the client, modifying it there, then writing it back. Changes are also sent while the storage is open.

TODO:
* https://bugs.meego.com/show_bug.cgi?id=18760 - "QtContacts-EDS + EDS: more flexible EBookView (UID only, only future changes)"

=== Suppress Unnecessary Data Transfer ===

EDS currently sends all data when a view is requested, which is unnecessary when the client is only interested in change notifications. An extension to the EDS query language could optionally suppresses that sending of existing data when the view is opened, to be implemented only for the local contacts and calendar backends. This would allow easier integration with the QContact API.

The Fremantle EDS fork also had a “freezable” (static) capability for libebook, which should probably be investigated. See, for instance: http://maemo.org/api_refs/5.0/5.0-final/libebook/EBookView.html#e-book-view-set-freezable

One possibility for adding the flag is to (ab)use the existing query language with a match for special fields:
* in the client: query = e_book_query_orv(e_book_query_field_exists("X-EDS-QUERY-FLAG-SKIP-DATA-DUMP"), <real query>, NULL);
* in the server: "if (query is OR-Query and first clause is exists("X-EDS-QUERY-FLAG-SKIP-DATA-DUMP")) { skip_data_dump = TRUE; query = second clause; }

=== Nofify with UIDs Only ===

EBookView emits signals when contacts are changed, added, or removed. But those signals sends lists of entire EContacts, containing the entire contact data, though we often have all the data already and just want to know which contacts have changed. This would allow easier integration with the QContact API, greatly improving performance.

[http://developer.gnome.org/libebook/stable/EBook.html#e-book-get-book-view e_book_get_book_view()] takes an [http://developer.gnome.org/libebook/stable/libebook-e-book-query.html#EBookQuery EBookQuery] which could be extended to take a new flag to make the EBookView send only the UID in the signals' EContacts. For instance, "X-EDS-QUERY-FLAG-NOTIFY-UID-ONLY". The UID is already stored separately for use as the primary key of the local-storage backend's BDB database, so no VCard parsing should be necessary to provide it.

Note that users of this feature should probably use just the base EVCard API on the provided EContact objects, to avoid extra processing and caching in the EContact code.

Note: We previously considered implementing e_book_get_book_view()'s requested fields parameter, but that changes the whole view, though we only really want to change what is sent for changed/new contacts.

Note: Chris discovered a dedicated e_book_get_contact_uids() API in the Fremantle EDS fork. The new API call was accepted upstream (in master) and [https://bugzilla.gnome.org/show_bug.cgi?id=651446#c11 backported as a patch] (used in the Meego package) for the gnome-2-32 branch, but this is not a full solution.

== Optional Parsing of ebook Data in Client ==

The EDS client-side APIs currently parse and (temporarily) store the parsed details of VCards. However, this is often unnecessary because APIs such as e_book_get_book_view() cause it to return the whole VCard anyway. There will be some performance gain from delaying VCard parsing until it is really needed by calls to functions such as e_contact_new_from_vcard().

libebook provides different functionality:
# exchange data as vCard with EDS daemon
# parse items and wrap them in a C API

QtContacts replaces the second part. An obvious performance improvement is to only do the second part if the additional C APIs are really called. The following calls then transfer data without any additional parsing in libebook:
* e_book_get_contact()/asynchronous read/view + e_vcard_to_string()
* e_contact_new_from_vcard() + e_book_add_contact()/e_book_commit_contact()/asynchronous storing

Note that libecal (for iCalendar) can already do this, though libebook can't yet.

TODO:
# add delayed parsing to libebook, triggered by API call which needs parsed data

PROBLEM:
# If delayed parsing fails, how can error be reported?
# Should delayed parsing be done always (e_vcard_to_string())) or only if requested? How would that be configured by a libebook user?

An existing (and in use) [http://build.meego.com/package/view_file?file=EBookView-Implement-delayed-vcard-parsing.patch&package=evolution-data-server&project=eds&srcmd5=977d2b885bed818cb616599749407e79 patch] might partly implement this already.

== Asynchronous API for storing items in libecal ==

libecal should allow the caller to request storing of data without blocking on processing of the data. It should notify the caller when the data has been stored, or when processing has failed.

These functions may require async versions:
* icalcomponent_new_from_string()
* e_cal_modify_object[_with_mod]
* e_cal_create_object()

Making libical asynchronous is almost impossible without going multithreaded. Let's focus on e_cal D-Bus communication first.

Rob started some work on this in the rbradford-wip-ecal-async-api branch. Perhaps you can reuse some of that work. It is not functional, but might be a source of inspiration or code fragments.

== More Efficient Access to Meta Data for Change Tracking ==

This was discussed a while ago: [[http://www.mail-archive.com/evolution-hackers@gnome.org/msg03029.html
[Evolution-hackers] concurrent modifications of items in GUI and EDS database]]

Data synchronization only needs to know which items exist (list of local IDs) and which revision of each item is currently stored. Revisions can be identified with a string which changes for each modification. Restoring an older revision should (but doesn't have to) restore the revision string.

For calendar, UID+RECURRENCE-ID and LAST-MODIFIED can be used. For contacts, UID + REV.

By comparing a stored list against the current one, changes can be detected without relying on journals in the daemon or comparison of time stamps. Journals are hard to maintain (how many of them, how long?). Comparison of time stamps has problems when changes are allowed while a sync runs and assumes that system time is linear (which is not always true).

As an additional use case, local ID + revision string could be used to prevent overwriting stale data in the server, as in HTTP PUT eTag checks.

TODO:

The Maemo Fremantle port of EDS, which is of course already open source, apparently has some solution for this, according to Mathias Hasselman, so that should be investigated first.

However, this should probably be made possible by:

# Extending the EDS query language with a flag that limits the delivered data to just:
## UID and REV for libebook (REV is in VCard 3.0)
## UID+RECURRENCE-ID+LAST-MODIFIED, for libecal
# Implementing these queries for the local contact and calendar backends. The query would be rejected, with an error, for other backends.

== Atomic Updates ==

Later, implement atomic changes as discussed in the mail thread: [[http://www.mail-archive.com/evolution-hackers@gnome.org/msg03029.html

== Quick check for "data changed" ==

The libebook and libical APIs should offer a way to discover if any data whatsoever has changed. This would, for instance, help SyncEvolution to avoid unnecessary detailed checking. This could just check the modification dates of local storage files, such as the BDB .db file, or .ics file, as long as we do not use those files to store permanent caches of parsed data. This would be much quicker than listing items for the whole database and comparing their revisions strings, as is currently necessary.

This would require small API additions, to check that the opaque revision strings are unchanged, such as

gchar* e_book_get_revision(EBook *book)
gboolean e_book_check_revision(EBook *book , const gchar* revision)

and

gchar* e_cal_get_revision(ECal *ecal)
gboolean e_cal_check_revision(ECal ecal, const gchar revision)

The semantic would be this:
* If the revision string is empty or different, then the content of the database '''might''' have changed. A more detailed check is necessary to determine if it has really changed. This will be the case for non-local storage.
* If it is non-empty and the same as before, the nothing has changed. No further check will be necessary.

== Contacts: Store PHOTO Data as Plain Files ==

Photos should be transferred over D-Bus via a filepath to a local file rather than transferring the actual photo data over D-Bus. The VCard format already allows this option (See the PHOTO property). We must be careful to keep these local files in sync and to remove them when contacts are removed, and to restrict access to the files from other processes.

There should be utility code available to libebook users to convert between the different representations.

Goals:
# Apps should be able to create photo files without ever passing the data to libebook.
# External PHOTO file and corresponding contact must be kept in sync:
** If a contact is deleted, the photo also needs to be removed.
** If a photo file is created and then storing the contact fails, who removes the file?
# Access to photo data must be limited to processes which have the right permissions.

Proposal:
* For each ~/.local/share/evolution/foo calendar, store photos in <photo path>=~/.local/share/evolution/foo/photos, with same permissions.
* New photo files are created with a [http://en.wikipedia.org/wiki/Universally_unique_identifier UUID] as base name and suffix according to the file content.
* If a contact is deleted and has a PHOTO;VALUE=uri:file://<photo path>/*, delete that file.
* For apps which want to create the photo file directly:
** Apps get a file descriptor and the path from a new libebook API.
** They own the file until the EDS server acknowledges the successful storing of a contact with a matching PHOTO;VALUE=uri:file property. If that fails, the app must remove the file.
* Add new API calls to libebook which split out resp. include data inline.
** Must track ownership of file after splitting it out and before storing - might never get stored!

Open:
* Should there be automatic garbage collection of stale photos in case something went wrong and a photo wasn't properly deleted?
* Trust TYPE=JPEG/PNG/... or look into file content to determine type?

== Future: Content Protection ==

Only authorized applications (= processes) with the necessary permissions should be able to read/write data. Exact permissions and mechanisms for verifying them to be defined as part of the (revised?) MeeGo architecture.

This can be added as follows:
# lock down file permissions
# add permission checks to EDS D-Bus servers

Currently not planned. Depends on the future MeeGo security infrastructure.

Architecture/planning/evolution-data-server/eds-improvements

2011-06-09T08:54:34Z

Murrayc: /* Return UIDs Only */

This page describes improvements that are desirable or even essential for good performance of Evolution Data Server (EDS) as PIM storage in MeeGo.

All of these changes should be developed for the evolution-data-server gnome-2-32 branch, as used by MeeGo 1.2, then ported to master for review by upstream maintainers. Most changes will not be accepted in upstream's gnome-2-32 branch, because is is stable, but Meego will apply them as patches to its package.

== Change Tracking ==

EDS is based on the model/view/controller principle: an application opens a database (calendar or addressbook) and requests a view with server-side filtering, then the application is sent:
# all data covered by the view
# the "all data sent" signal
# changes (add/update/remove) while the view exists

Applications modify data with asynchronous requests (controller) and keep showing the old data until the change notification comes in. The exact changes made to the data are determined by the server (for example when meeting invitations are sent).

QtContacts and KCalCore do not have such a strict model. With QtContacts the app simply loads and stores contacts and is responsible for maintaining its own view. Changes are sent for changes made after the database was opened. KCalCore is based on the concept of loading a calendar (partially) into the client, modifying it there, then writing it back. Changes are also sent while the storage is open.

TODO:
* https://bugs.meego.com/show_bug.cgi?id=18760 - "QtContacts-EDS + EDS: more flexible EBookView (UID only, only future changes)"

=== Suppress Unnecessary Data Transfer ===

EDS currently sends all data when a view is requested, which is unnecessary when the client is only interested in change notifications. An extension to the EDS query language could optionally suppresses that sending of existing data when the view is opened, to be implemented only for the local contacts and calendar backends. This would allow easier integration with the QContact API.

The Fremantle EDS fork also had a “freezable” (static) capability for libebook, which should probably be investigated. See, for instance: http://maemo.org/api_refs/5.0/5.0-final/libebook/EBookView.html#e-book-view-set-freezable

One possibility for adding the flag is to (ab)use the existing query language with a match for special fields:
* in the client: query = e_book_query_orv(e_book_query_field_exists("X-EDS-QUERY-FLAG-SKIP-DATA-DUMP"), <real query>, NULL);
* in the server: "if (query is OR-Query and first clause is exists("X-EDS-QUERY-FLAG-SKIP-DATA-DUMP")) { skip_data_dump = TRUE; query = second clause; }

=== Return UIDs Only ===

The e_book_get_book_view() function takes a list of requested fields, to avoid returning unnecessary data, but this is apparently ignored for the local contacts backend. It would anyway require extra parsing and generation of smaller VCards, losing some possible advantages from delayed parsing of VCards.

However, the UID is already stored separately for use as the primary key of the BDB database, so it would be worthwhile to support that particular requested field. This would allow easier integration with the QContact API. It would also offer much better performance in cases where the client already has the data but just wants to discover what subset of that data meets a certain criteria.

Chris discovered a dedicated API for this in the Fremantle EDS fork. The new API call was accepted upstream and [https://bugzilla.gnome.org/show_bug.cgi?id=651446#c11 backported as a patch] (used in the Meego package) for the gnome-2-32, so no further work is needed?

== Optional Parsing of ebook Data in Client ==

The EDS client-side APIs currently parse and (temporarily) store the parsed details of VCards. However, this is often unnecessary because APIs such as e_book_get_book_view() cause it to return the whole VCard anyway. There will be some performance gain from delaying VCard parsing until it is really needed by calls to functions such as e_contact_new_from_vcard().

libebook provides different functionality:
# exchange data as vCard with EDS daemon
# parse items and wrap them in a C API

QtContacts replaces the second part. An obvious performance improvement is to only do the second part if the additional C APIs are really called. The following calls then transfer data without any additional parsing in libebook:
* e_book_get_contact()/asynchronous read/view + e_vcard_to_string()
* e_contact_new_from_vcard() + e_book_add_contact()/e_book_commit_contact()/asynchronous storing

Note that libecal (for iCalendar) can already do this, though libebook can't yet.

TODO:
# add delayed parsing to libebook, triggered by API call which needs parsed data

PROBLEM:
# If delayed parsing fails, how can error be reported?
# Should delayed parsing be done always (e_vcard_to_string())) or only if requested? How would that be configured by a libebook user?

An existing (and in use) [http://build.meego.com/package/view_file?file=EBookView-Implement-delayed-vcard-parsing.patch&package=evolution-data-server&project=eds&srcmd5=977d2b885bed818cb616599749407e79 patch] might partly implement this already.

== Asynchronous API for storing items in libecal ==

libecal should allow the caller to request storing of data without blocking on processing of the data. It should notify the caller when the data has been stored, or when processing has failed.

These functions may require async versions:
* icalcomponent_new_from_string()
* e_cal_modify_object[_with_mod]
* e_cal_create_object()

Making libical asynchronous is almost impossible without going multithreaded. Let's focus on e_cal D-Bus communication first.

Rob started some work on this in the rbradford-wip-ecal-async-api branch. Perhaps you can reuse some of that work. It is not functional, but might be a source of inspiration or code fragments.

== More Efficient Access to Meta Data for Change Tracking ==

This was discussed a while ago: [[http://www.mail-archive.com/evolution-hackers@gnome.org/msg03029.html
[Evolution-hackers] concurrent modifications of items in GUI and EDS database]]

Data synchronization only needs to know which items exist (list of local IDs) and which revision of each item is currently stored. Revisions can be identified with a string which changes for each modification. Restoring an older revision should (but doesn't have to) restore the revision string.

For calendar, UID+RECURRENCE-ID and LAST-MODIFIED can be used. For contacts, UID + REV.

By comparing a stored list against the current one, changes can be detected without relying on journals in the daemon or comparison of time stamps. Journals are hard to maintain (how many of them, how long?). Comparison of time stamps has problems when changes are allowed while a sync runs and assumes that system time is linear (which is not always true).

As an additional use case, local ID + revision string could be used to prevent overwriting stale data in the server, as in HTTP PUT eTag checks.

TODO:

The Maemo Fremantle port of EDS, which is of course already open source, apparently has some solution for this, according to Mathias Hasselman, so that should be investigated first.

However, this should probably be made possible by:

# Extending the EDS query language with a flag that limits the delivered data to just:
## UID and REV for libebook (REV is in VCard 3.0)
## UID+RECURRENCE-ID+LAST-MODIFIED, for libecal
# Implementing these queries for the local contact and calendar backends. The query would be rejected, with an error, for other backends.

== Atomic Updates ==

Later, implement atomic changes as discussed in the mail thread: [[http://www.mail-archive.com/evolution-hackers@gnome.org/msg03029.html

== Quick check for "data changed" ==

The libebook and libical APIs should offer a way to discover if any data whatsoever has changed. This would, for instance, help SyncEvolution to avoid unnecessary detailed checking. This could just check the modification dates of local storage files, such as the BDB .db file, or .ics file, as long as we do not use those files to store permanent caches of parsed data. This would be much quicker than listing items for the whole database and comparing their revisions strings, as is currently necessary.

This would require small API additions, to check that the opaque revision strings are unchanged, such as

gchar* e_book_get_revision(EBook *book)
gboolean e_book_check_revision(EBook *book , const gchar* revision)

and

gchar* e_cal_get_revision(ECal *ecal)
gboolean e_cal_check_revision(ECal ecal, const gchar revision)

The semantic would be this:
* If the revision string is empty or different, then the content of the database '''might''' have changed. A more detailed check is necessary to determine if it has really changed. This will be the case for non-local storage.
* If it is non-empty and the same as before, the nothing has changed. No further check will be necessary.

== Contacts: Store PHOTO Data as Plain Files ==

Photos should be transferred over D-Bus via a filepath to a local file rather than transferring the actual photo data over D-Bus. The VCard format already allows this option (See the PHOTO property). We must be careful to keep these local files in sync and to remove them when contacts are removed, and to restrict access to the files from other processes.

There should be utility code available to libebook users to convert between the different representations.

Goals:
# Apps should be able to create photo files without ever passing the data to libebook.
# External PHOTO file and corresponding contact must be kept in sync:
** If a contact is deleted, the photo also needs to be removed.
** If a photo file is created and then storing the contact fails, who removes the file?
# Access to photo data must be limited to processes which have the right permissions.

Proposal:
* For each ~/.local/share/evolution/foo calendar, store photos in <photo path>=~/.local/share/evolution/foo/photos, with same permissions.
* New photo files are created with a [http://en.wikipedia.org/wiki/Universally_unique_identifier UUID] as base name and suffix according to the file content.
* If a contact is deleted and has a PHOTO;VALUE=uri:file://<photo path>/*, delete that file.
* For apps which want to create the photo file directly:
** Apps get a file descriptor and the path from a new libebook API.
** They own the file until the EDS server acknowledges the successful storing of a contact with a matching PHOTO;VALUE=uri:file property. If that fails, the app must remove the file.
* Add new API calls to libebook which split out resp. include data inline.
** Must track ownership of file after splitting it out and before storing - might never get stored!

Open:
* Should there be automatic garbage collection of stale photos in case something went wrong and a photo wasn't properly deleted?
* Trust TYPE=JPEG/PNG/... or look into file content to determine type?

== Future: Content Protection ==

Only authorized applications (= processes) with the necessary permissions should be able to read/write data. Exact permissions and mechanisms for verifying them to be defined as part of the (revised?) MeeGo architecture.

This can be added as follows:
# lock down file permissions
# add permission checks to EDS D-Bus servers

Currently not planned. Depends on the future MeeGo security infrastructure.

Architecture/planning/evolution-data-server/eds-improvements

2011-06-09T08:52:59Z

Murrayc: /* Return UIDs Only */

This page describes improvements that are desirable or even essential for good performance of Evolution Data Server (EDS) as PIM storage in MeeGo.

All of these changes should be developed for the evolution-data-server gnome-2-32 branch, as used by MeeGo 1.2, then ported to master for review by upstream maintainers. Most changes will not be accepted in upstream's gnome-2-32 branch, because is is stable, but Meego will apply them as patches to its package.

== Change Tracking ==

EDS is based on the model/view/controller principle: an application opens a database (calendar or addressbook) and requests a view with server-side filtering, then the application is sent:
# all data covered by the view
# the "all data sent" signal
# changes (add/update/remove) while the view exists

Applications modify data with asynchronous requests (controller) and keep showing the old data until the change notification comes in. The exact changes made to the data are determined by the server (for example when meeting invitations are sent).

QtContacts and KCalCore do not have such a strict model. With QtContacts the app simply loads and stores contacts and is responsible for maintaining its own view. Changes are sent for changes made after the database was opened. KCalCore is based on the concept of loading a calendar (partially) into the client, modifying it there, then writing it back. Changes are also sent while the storage is open.

TODO:
* https://bugs.meego.com/show_bug.cgi?id=18760 - "QtContacts-EDS + EDS: more flexible EBookView (UID only, only future changes)"

=== Suppress Unnecessary Data Transfer ===

EDS currently sends all data when a view is requested, which is unnecessary when the client is only interested in change notifications. An extension to the EDS query language could optionally suppresses that sending of existing data when the view is opened, to be implemented only for the local contacts and calendar backends. This would allow easier integration with the QContact API.

The Fremantle EDS fork also had a “freezable” (static) capability for libebook, which should probably be investigated. See, for instance: http://maemo.org/api_refs/5.0/5.0-final/libebook/EBookView.html#e-book-view-set-freezable

One possibility for adding the flag is to (ab)use the existing query language with a match for special fields:
* in the client: query = e_book_query_orv(e_book_query_field_exists("X-EDS-QUERY-FLAG-SKIP-DATA-DUMP"), <real query>, NULL);
* in the server: "if (query is OR-Query and first clause is exists("X-EDS-QUERY-FLAG-SKIP-DATA-DUMP")) { skip_data_dump = TRUE; query = second clause; }

=== Return UIDs Only ===

The e_book_get_book_view() function takes a list of requested fields, to avoid returning unnecessary data, but this is apparently ignored for the local contacts backend. It would anyway require extra parsing and generation of smaller VCards, losing some possible advantages from delayed parsing of VCards.

However, the UID is already stored separately for use as the primary key of the BDB database, so it would be worthwhile to support that particular requested field. This would allow easier integration with the QContact API. It would also offer much better performance in cases where the client already has the data but just wants to discover what subset of that data meets a certain criteria.

Chris discovered a dedicated API for this in the Fremantle EDS fork. The new API call was accepted upstream and [https://bugzilla.gnome.org/show_bug.cgi?id=651446#c11|backported as a patch] (used in the Meego package) for the gnome-2-32, so no further work is needed?

== Optional Parsing of ebook Data in Client ==

The EDS client-side APIs currently parse and (temporarily) store the parsed details of VCards. However, this is often unnecessary because APIs such as e_book_get_book_view() cause it to return the whole VCard anyway. There will be some performance gain from delaying VCard parsing until it is really needed by calls to functions such as e_contact_new_from_vcard().

libebook provides different functionality:
# exchange data as vCard with EDS daemon
# parse items and wrap them in a C API

QtContacts replaces the second part. An obvious performance improvement is to only do the second part if the additional C APIs are really called. The following calls then transfer data without any additional parsing in libebook:
* e_book_get_contact()/asynchronous read/view + e_vcard_to_string()
* e_contact_new_from_vcard() + e_book_add_contact()/e_book_commit_contact()/asynchronous storing

Note that libecal (for iCalendar) can already do this, though libebook can't yet.

TODO:
# add delayed parsing to libebook, triggered by API call which needs parsed data

PROBLEM:
# If delayed parsing fails, how can error be reported?
# Should delayed parsing be done always (e_vcard_to_string())) or only if requested? How would that be configured by a libebook user?

An existing (and in use) [http://build.meego.com/package/view_file?file=EBookView-Implement-delayed-vcard-parsing.patch&package=evolution-data-server&project=eds&srcmd5=977d2b885bed818cb616599749407e79 patch] might partly implement this already.

== Asynchronous API for storing items in libecal ==

libecal should allow the caller to request storing of data without blocking on processing of the data. It should notify the caller when the data has been stored, or when processing has failed.

These functions may require async versions:
* icalcomponent_new_from_string()
* e_cal_modify_object[_with_mod]
* e_cal_create_object()

Making libical asynchronous is almost impossible without going multithreaded. Let's focus on e_cal D-Bus communication first.

Rob started some work on this in the rbradford-wip-ecal-async-api branch. Perhaps you can reuse some of that work. It is not functional, but might be a source of inspiration or code fragments.

== More Efficient Access to Meta Data for Change Tracking ==

This was discussed a while ago: [[http://www.mail-archive.com/evolution-hackers@gnome.org/msg03029.html
[Evolution-hackers] concurrent modifications of items in GUI and EDS database]]

Data synchronization only needs to know which items exist (list of local IDs) and which revision of each item is currently stored. Revisions can be identified with a string which changes for each modification. Restoring an older revision should (but doesn't have to) restore the revision string.

For calendar, UID+RECURRENCE-ID and LAST-MODIFIED can be used. For contacts, UID + REV.

By comparing a stored list against the current one, changes can be detected without relying on journals in the daemon or comparison of time stamps. Journals are hard to maintain (how many of them, how long?). Comparison of time stamps has problems when changes are allowed while a sync runs and assumes that system time is linear (which is not always true).

As an additional use case, local ID + revision string could be used to prevent overwriting stale data in the server, as in HTTP PUT eTag checks.

TODO:

The Maemo Fremantle port of EDS, which is of course already open source, apparently has some solution for this, according to Mathias Hasselman, so that should be investigated first.

However, this should probably be made possible by:

# Extending the EDS query language with a flag that limits the delivered data to just:
## UID and REV for libebook (REV is in VCard 3.0)
## UID+RECURRENCE-ID+LAST-MODIFIED, for libecal
# Implementing these queries for the local contact and calendar backends. The query would be rejected, with an error, for other backends.

== Atomic Updates ==

Later, implement atomic changes as discussed in the mail thread: [[http://www.mail-archive.com/evolution-hackers@gnome.org/msg03029.html

== Quick check for "data changed" ==

The libebook and libical APIs should offer a way to discover if any data whatsoever has changed. This would, for instance, help SyncEvolution to avoid unnecessary detailed checking. This could just check the modification dates of local storage files, such as the BDB .db file, or .ics file, as long as we do not use those files to store permanent caches of parsed data. This would be much quicker than listing items for the whole database and comparing their revisions strings, as is currently necessary.

This would require small API additions, to check that the opaque revision strings are unchanged, such as

gchar* e_book_get_revision(EBook *book)
gboolean e_book_check_revision(EBook *book , const gchar* revision)

and

gchar* e_cal_get_revision(ECal *ecal)
gboolean e_cal_check_revision(ECal ecal, const gchar revision)

The semantic would be this:
* If the revision string is empty or different, then the content of the database '''might''' have changed. A more detailed check is necessary to determine if it has really changed. This will be the case for non-local storage.
* If it is non-empty and the same as before, the nothing has changed. No further check will be necessary.

== Contacts: Store PHOTO Data as Plain Files ==

Photos should be transferred over D-Bus via a filepath to a local file rather than transferring the actual photo data over D-Bus. The VCard format already allows this option (See the PHOTO property). We must be careful to keep these local files in sync and to remove them when contacts are removed, and to restrict access to the files from other processes.

There should be utility code available to libebook users to convert between the different representations.

Goals:
# Apps should be able to create photo files without ever passing the data to libebook.
# External PHOTO file and corresponding contact must be kept in sync:
** If a contact is deleted, the photo also needs to be removed.
** If a photo file is created and then storing the contact fails, who removes the file?
# Access to photo data must be limited to processes which have the right permissions.

Proposal:
* For each ~/.local/share/evolution/foo calendar, store photos in <photo path>=~/.local/share/evolution/foo/photos, with same permissions.
* New photo files are created with a [http://en.wikipedia.org/wiki/Universally_unique_identifier UUID] as base name and suffix according to the file content.
* If a contact is deleted and has a PHOTO;VALUE=uri:file://<photo path>/*, delete that file.
* For apps which want to create the photo file directly:
** Apps get a file descriptor and the path from a new libebook API.
** They own the file until the EDS server acknowledges the successful storing of a contact with a matching PHOTO;VALUE=uri:file property. If that fails, the app must remove the file.
* Add new API calls to libebook which split out resp. include data inline.
** Must track ownership of file after splitting it out and before storing - might never get stored!

Open:
* Should there be automatic garbage collection of stale photos in case something went wrong and a photo wasn't properly deleted?
* Trust TYPE=JPEG/PNG/... or look into file content to determine type?

== Future: Content Protection ==

Only authorized applications (= processes) with the necessary permissions should be able to read/write data. Exact permissions and mechanisms for verifying them to be defined as part of the (revised?) MeeGo architecture.

This can be added as follows:
# lock down file permissions
# add permission checks to EDS D-Bus servers

Currently not planned. Depends on the future MeeGo security infrastructure.

Architecture/planning/evolution-data-server/eds-improvements

2011-06-09T08:39:25Z

Murrayc: A change of plan about what branch to target first.

This page describes improvements that are desirable or even essential for good performance of Evolution Data Server (EDS) as PIM storage in MeeGo.

All of these changes should be developed for the evolution-data-server gnome-2-32 branch, as used by MeeGo 1.2, then ported to master for review by upstream maintainers. Most changes will not be accepted in upstream's gnome-2-32 branch, because is is stable, but Meego will apply them as patches to its package.

== Change Tracking ==

EDS is based on the model/view/controller principle: an application opens a database (calendar or addressbook) and requests a view with server-side filtering, then the application is sent:
# all data covered by the view
# the "all data sent" signal
# changes (add/update/remove) while the view exists

Applications modify data with asynchronous requests (controller) and keep showing the old data until the change notification comes in. The exact changes made to the data are determined by the server (for example when meeting invitations are sent).

QtContacts and KCalCore do not have such a strict model. With QtContacts the app simply loads and stores contacts and is responsible for maintaining its own view. Changes are sent for changes made after the database was opened. KCalCore is based on the concept of loading a calendar (partially) into the client, modifying it there, then writing it back. Changes are also sent while the storage is open.

TODO:
* https://bugs.meego.com/show_bug.cgi?id=18760 - "QtContacts-EDS + EDS: more flexible EBookView (UID only, only future changes)"

=== Suppress Unnecessary Data Transfer ===

EDS currently sends all data when a view is requested, which is unnecessary when the client is only interested in change notifications. An extension to the EDS query language could optionally suppresses that sending of existing data when the view is opened, to be implemented only for the local contacts and calendar backends. This would allow easier integration with the QContact API.

The Fremantle EDS fork also had a “freezable” (static) capability for libebook, which should probably be investigated. See, for instance: http://maemo.org/api_refs/5.0/5.0-final/libebook/EBookView.html#e-book-view-set-freezable

One possibility for adding the flag is to (ab)use the existing query language with a match for special fields:
* in the client: query = e_book_query_orv(e_book_query_field_exists("X-EDS-QUERY-FLAG-SKIP-DATA-DUMP"), <real query>, NULL);
* in the server: "if (query is OR-Query and first clause is exists("X-EDS-QUERY-FLAG-SKIP-DATA-DUMP")) { skip_data_dump = TRUE; query = second clause; }

=== Return UIDs Only ===

The e_book_get_book_view() function takes a list of requested fields, to avoid returning unnecessary data, but this is apparently ignored for the local contacts backend. It would anyway require extra parsing and generation of smaller VCards, losing some possible advantages from delayed parsing of VCards.

However, the UID is already stored separately for use as the primary key of the BDB database, so it would be worthwhile to support that particular requested field. This would allow easier integration with the QContact API. It would also offer much better performance in cases where the client already has the data but just wants to discover what subset of that data meets a certain criteria.

Chris discovered a dedicated API for this in the Fremantle EDS fork. The new API call was accepted upstream and [https://bugzilla.gnome.org/show_bug.cgi?id=651446#c11 backported as a patch] (used in the Meego package) for the gnome-2-32, so no further work is needed?

== Optional Parsing of ebook Data in Client ==

The EDS client-side APIs currently parse and (temporarily) store the parsed details of VCards. However, this is often unnecessary because APIs such as e_book_get_book_view() cause it to return the whole VCard anyway. There will be some performance gain from delaying VCard parsing until it is really needed by calls to functions such as e_contact_new_from_vcard().

libebook provides different functionality:
# exchange data as vCard with EDS daemon
# parse items and wrap them in a C API

QtContacts replaces the second part. An obvious performance improvement is to only do the second part if the additional C APIs are really called. The following calls then transfer data without any additional parsing in libebook:
* e_book_get_contact()/asynchronous read/view + e_vcard_to_string()
* e_contact_new_from_vcard() + e_book_add_contact()/e_book_commit_contact()/asynchronous storing

Note that libecal (for iCalendar) can already do this, though libebook can't yet.

TODO:
# add delayed parsing to libebook, triggered by API call which needs parsed data

PROBLEM:
# If delayed parsing fails, how can error be reported?
# Should delayed parsing be done always (e_vcard_to_string())) or only if requested? How would that be configured by a libebook user?

An existing (and in use) [http://build.meego.com/package/view_file?file=EBookView-Implement-delayed-vcard-parsing.patch&package=evolution-data-server&project=eds&srcmd5=977d2b885bed818cb616599749407e79 patch] might partly implement this already.

== Asynchronous API for storing items in libecal ==

libecal should allow the caller to request storing of data without blocking on processing of the data. It should notify the caller when the data has been stored, or when processing has failed.

These functions may require async versions:
* icalcomponent_new_from_string()
* e_cal_modify_object[_with_mod]
* e_cal_create_object()

Making libical asynchronous is almost impossible without going multithreaded. Let's focus on e_cal D-Bus communication first.

Rob started some work on this in the rbradford-wip-ecal-async-api branch. Perhaps you can reuse some of that work. It is not functional, but might be a source of inspiration or code fragments.

== More Efficient Access to Meta Data for Change Tracking ==

This was discussed a while ago: [[http://www.mail-archive.com/evolution-hackers@gnome.org/msg03029.html
[Evolution-hackers] concurrent modifications of items in GUI and EDS database]]

Data synchronization only needs to know which items exist (list of local IDs) and which revision of each item is currently stored. Revisions can be identified with a string which changes for each modification. Restoring an older revision should (but doesn't have to) restore the revision string.

For calendar, UID+RECURRENCE-ID and LAST-MODIFIED can be used. For contacts, UID + REV.

By comparing a stored list against the current one, changes can be detected without relying on journals in the daemon or comparison of time stamps. Journals are hard to maintain (how many of them, how long?). Comparison of time stamps has problems when changes are allowed while a sync runs and assumes that system time is linear (which is not always true).

As an additional use case, local ID + revision string could be used to prevent overwriting stale data in the server, as in HTTP PUT eTag checks.

TODO:

The Maemo Fremantle port of EDS, which is of course already open source, apparently has some solution for this, according to Mathias Hasselman, so that should be investigated first.

However, this should probably be made possible by:

# Extending the EDS query language with a flag that limits the delivered data to just:
## UID and REV for libebook (REV is in VCard 3.0)
## UID+RECURRENCE-ID+LAST-MODIFIED, for libecal
# Implementing these queries for the local contact and calendar backends. The query would be rejected, with an error, for other backends.

== Atomic Updates ==

Later, implement atomic changes as discussed in the mail thread: [[http://www.mail-archive.com/evolution-hackers@gnome.org/msg03029.html

== Quick check for "data changed" ==

The libebook and libical APIs should offer a way to discover if any data whatsoever has changed. This would, for instance, help SyncEvolution to avoid unnecessary detailed checking. This could just check the modification dates of local storage files, such as the BDB .db file, or .ics file, as long as we do not use those files to store permanent caches of parsed data. This would be much quicker than listing items for the whole database and comparing their revisions strings, as is currently necessary.

This would require small API additions, to check that the opaque revision strings are unchanged, such as

gchar* e_book_get_revision(EBook *book)
gboolean e_book_check_revision(EBook *book , const gchar* revision)

and

gchar* e_cal_get_revision(ECal *ecal)
gboolean e_cal_check_revision(ECal ecal, const gchar revision)

The semantic would be this:
* If the revision string is empty or different, then the content of the database '''might''' have changed. A more detailed check is necessary to determine if it has really changed. This will be the case for non-local storage.
* If it is non-empty and the same as before, the nothing has changed. No further check will be necessary.

== Contacts: Store PHOTO Data as Plain Files ==

Photos should be transferred over D-Bus via a filepath to a local file rather than transferring the actual photo data over D-Bus. The VCard format already allows this option (See the PHOTO property). We must be careful to keep these local files in sync and to remove them when contacts are removed, and to restrict access to the files from other processes.

There should be utility code available to libebook users to convert between the different representations.

Goals:
# Apps should be able to create photo files without ever passing the data to libebook.
# External PHOTO file and corresponding contact must be kept in sync:
** If a contact is deleted, the photo also needs to be removed.
** If a photo file is created and then storing the contact fails, who removes the file?
# Access to photo data must be limited to processes which have the right permissions.

Proposal:
* For each ~/.local/share/evolution/foo calendar, store photos in <photo path>=~/.local/share/evolution/foo/photos, with same permissions.
* New photo files are created with a [http://en.wikipedia.org/wiki/Universally_unique_identifier UUID] as base name and suffix according to the file content.
* If a contact is deleted and has a PHOTO;VALUE=uri:file://<photo path>/*, delete that file.
* For apps which want to create the photo file directly:
** Apps get a file descriptor and the path from a new libebook API.
** They own the file until the EDS server acknowledges the successful storing of a contact with a matching PHOTO;VALUE=uri:file property. If that fails, the app must remove the file.
* Add new API calls to libebook which split out resp. include data inline.
** Must track ownership of file after splitting it out and before storing - might never get stored!

Open:
* Should there be automatic garbage collection of stale photos in case something went wrong and a photo wasn't properly deleted?
* Trust TYPE=JPEG/PNG/... or look into file content to determine type?

== Future: Content Protection ==

Only authorized applications (= processes) with the necessary permissions should be able to read/write data. Exact permissions and mechanisms for verifying them to be defined as part of the (revised?) MeeGo architecture.

This can be added as follows:
# lock down file permissions
# add permission checks to EDS D-Bus servers

Currently not planned. Depends on the future MeeGo security infrastructure.

Architecture/planning/evolution-data-server/eds-improvements

2011-06-08T12:48:54Z

Murrayc: /* Optional Parsing of ebook Data in Client */

This page describes improvements that are desirable or even essential for good performance of Evolution Data Server (EDS) as PIM storage in MeeGo.

All of these changes should be developed for the EDS master branch, reviewed by upstream maintainers and then back-ported to EDS 2.32, the version in MeeGo.

== Change Tracking ==

EDS is based on the model/view/controller principle: an application opens a database (calendar or addressbook) and requests a view with server-side filtering, then the application is sent:
# all data covered by the view
# the "all data sent" signal
# changes (add/update/remove) while the view exists

Applications modify data with asynchronous requests (controller) and keep showing the old data until the change notification comes in. The exact changes made to the data are determined by the server (for example when meeting invitations are sent).

QtContacts and KCalCore do not have such a strict model. With QtContacts the app simply loads and stores contacts and is responsible for maintaining its own view. Changes are sent for changes made after the database was opened. KCalCore is based on the concept of loading a calendar (partially) into the client, modifying it there, then writing it back. Changes are also sent while the storage is open.

TODO:

=== Suppress Unnecessary Data Transfer ===

EDS currently sends all data when a view is requested, which is unnecessary when the client is only interested in change notifications. An extension to the EDS query language could optionally suppresses that sending of existing data when the view is opened, to be implemented only for the local contacts and calendar backends. This would allow easier integration with the QContact API.

The Fremantle EDS fork also had a “freezable” (static) capability for libebook, which should probably be investigated. See, for instance: http://maemo.org/api_refs/5.0/5.0-final/libebook/EBookView.html#e-book-view-set-freezable

One possibility for adding the flag is to (ab)use the existing query language with a match for special fields:
* in the client: query = e_book_query_orv(e_book_query_field_exists("X-EDS-QUERY-FLAG-SKIP-DATA-DUMP"), <real query>, NULL);
* in the server: "if (query is OR-Query and first clause is exists("X-EDS-QUERY-FLAG-SKIP-DATA-DUMP")) { skip_data_dump = TRUE; query = second clause; }

=== Return UIDs Only ===

The e_book_get_book_view() function takes a list of requested fields, to avoid returning unnecessary data, but this is apparently ignored for the local contacts backend. It would anyway require extra parsing and generation of smaller VCards, losing some possible advantages from delayed parsing of VCards.

However, the UID is already stored separately for use as the primary key of the BDB database, so it would be worthwhile to support that particular requested field. This would allow easier integration with the QContact API. It would also offer much better performance in cases where the client already has the data but just wants to discover what subset of that data meets a certain criteria.

Chris discovered a dedicated API for this in the Fremantle EDS fork. The new API call was accepted upstream and [https://bugzilla.gnome.org/show_bug.cgi?id=651446#c11 backported as a patch] (used in the Meego package) for the gnome-2-32, so no further work is needed?

== Optional Parsing of ebook Data in Client ==

The EDS client-side APIs currently parse and (temporarily) store the parsed details of VCards. However, this is often unnecessary because APIs such as e_book_get_book_view() cause it to return the whole VCard anyway. There will be some performance gain from delaying VCard parsing until it is really needed by calls to functions such as e_contact_new_from_vcard().

libebook provides different functionality:
# exchange data as vCard with EDS daemon
# parse items and wrap them in a C API

QtContacts replaces the second part. An obvious performance improvement is to only do the second part if the additional C APIs are really called. The following calls then transfer data without any additional parsing in libebook:
* e_book_get_contact()/asynchronous read/view + e_vcard_to_string()
* e_contact_new_from_vcard() + e_book_add_contact()/e_book_commit_contact()/asynchronous storing

Note that libecal (for iCalendar) can already do this, though libebook can't yet.

TODO:
# add delayed parsing to libebook, triggered by API call which needs parsed data

PROBLEM:
# If delayed parsing fails, how can error be reported?
# Should delayed parsing be done always (e_vcard_to_string())) or only if requested? How would that be configured by a libebook user?

An existing (and in use) [http://build.meego.com/package/view_file?file=EBookView-Implement-delayed-vcard-parsing.patch&package=evolution-data-server&project=eds&srcmd5=977d2b885bed818cb616599749407e79 patch] might partly implement this already.

== Asynchronous API for storing items in libecal ==

libecal should allow the caller to request storing of data without blocking on processing of the data. It should notify the caller when the data has been stored, or when processing has failed.

These functions may require async versions:
* icalcomponent_new_from_string()
* e_cal_modify_object[_with_mod]
* e_cal_create_object()

== More Efficient Access to Meta Data for Change Tracking ==

This was discussed a while ago: [[http://www.mail-archive.com/evolution-hackers@gnome.org/msg03029.html
[Evolution-hackers] concurrent modifications of items in GUI and EDS database]]

Data synchronization only needs to know which items exist (list of local IDs) and which revision of each item is currently stored. Revisions can be identified with a string which changes for each modification. Restoring an older revision should (but doesn't have to) restore the revision string.

For calendar, UID+RECURRENCE-ID and LAST-MODIFIED can be used. For contacts, UID + REV.

By comparing a stored list against the current one, changes can be detected without relying on journals in the daemon or comparison of time stamps. Journals are hard to maintain (how many of them, how long?). Comparison of time stamps has problems when changes are allowed while a sync runs and assumes that system time is linear (which is not always true).

As an additional use case, local ID + revision string could be used to prevent overwriting stale data in the server, as in HTTP PUT eTag checks.

TODO:

The Maemo Fremantle port of EDS, which is of course already open source, apparently has some solution for this, according to Mathias Hasselman, so that should be investigated first.

However, this should probably be made possible by:

# Extending the EDS query language with a flag that limits the delivered data to just:
## UID and REV for libebook (REV is in VCard 3.0)
## UID+RECURRENCE-ID+LAST-MODIFIED, for libecal
# Implementing these queries for the local contact and calendar backends. The query would be rejected, with an error, for other backends.

== Atomic Updates ==

Later, implement atomic changes as discussed in the mail thread: [[http://www.mail-archive.com/evolution-hackers@gnome.org/msg03029.html

== Quick check for "data changed" ==

The libebook and libical APIs should offer a way to discover if any data whatsoever has changed. This would, for instance, help SyncEvolution to avoid unnecessary detailed checking. This could just check the modification dates of local storage files, such as the BDB .db file, or .ics file, as long as we do not use those files to store permanent caches of parsed data. This would be much quicker than listing items for the whole database and comparing their revisions strings, as is currently necessary.

This would require small API additions, to check that the opaque revision strings are unchanged, such as

gchar* e_book_get_revision(EBook *book)
gboolean e_book_check_revision(EBook *book , const gchar* revision)

and

gchar* e_cal_get_revision(ECal *ecal)
gboolean e_cal_check_revision(ECal ecal, const gchar revision)

The semantic would be this:
* If the revision string is empty or different, then the content of the database '''might''' have changed. A more detailed check is necessary to determine if it has really changed. This will be the case for non-local storage.
* If it is non-empty and the same as before, the nothing has changed. No further check will be necessary.

== Contacts: Store PHOTO Data as Plain Files ==

Photos should be transferred over D-Bus via a filepath to a local file rather than transferring the actual photo data over D-Bus. The VCard format already allows this option (See the PHOTO property). We must be careful to keep these local files in sync and to remove them when contacts are removed, and to restrict access to the files from other processes.

There should be utility code available to libebook users to convert between the different representations.

Goals:
# Apps should be able to create photo files without ever passing the data to libebook.
# External PHOTO file and corresponding contact must be kept in sync:
** If a contact is deleted, the photo also needs to be removed.
** If a photo file is created and then storing the contact fails, who removes the file?
# Access to photo data must be limited to processes which have the right permissions.

Proposal:
* For each ~/.local/share/evolution/foo calendar, store photos in <photo path>=~/.local/share/evolution/foo/photos, with same permissions.
* New photo files are created with a [http://en.wikipedia.org/wiki/Universally_unique_identifier UUID] as base name and suffix according to the file content.
* If a contact is deleted and has a PHOTO;VALUE=uri:file://<photo path>/*, delete that file.
* For apps which want to create the photo file directly:
** Apps get a file descriptor and the path from a new libebook API.
** They own the file until the EDS server acknowledges the successful storing of a contact with a matching PHOTO;VALUE=uri:file property. If that fails, the app must remove the file.
* Add new API calls to libebook which split out resp. include data inline.
** Must track ownership of file after splitting it out and before storing - might never get stored!

Open:
* Should there be automatic garbage collection of stale photos in case something went wrong and a photo wasn't properly deleted?
* Trust TYPE=JPEG/PNG/... or look into file content to determine type?

== Future: Content Protection ==

Only authorized applications (= processes) with the necessary permissions should be able to read/write data. Exact permissions and mechanisms for verifying them to be defined as part of the (revised?) MeeGo architecture.

This can be added as follows:
# lock down file permissions
# add permission checks to EDS D-Bus servers

Currently not planned. Depends on the future MeeGo security infrastructure.

Architecture/planning/evolution-data-server/eds-improvements

2011-06-08T12:41:05Z

Murrayc: /* Return UIDs Only: Link to the patch that's already in use.

This page describes improvements that are desirable or even essential for good performance of Evolution Data Server (EDS) as PIM storage in MeeGo.

All of these changes should be developed for the EDS master branch, reviewed by upstream maintainers and then back-ported to EDS 2.32, the version in MeeGo.

== Change Tracking ==

EDS is based on the model/view/controller principle: an application opens a database (calendar or addressbook) and requests a view with server-side filtering, then the application is sent:
# all data covered by the view
# the "all data sent" signal
# changes (add/update/remove) while the view exists

Applications modify data with asynchronous requests (controller) and keep showing the old data until the change notification comes in. The exact changes made to the data are determined by the server (for example when meeting invitations are sent).

QtContacts and KCalCore do not have such a strict model. With QtContacts the app simply loads and stores contacts and is responsible for maintaining its own view. Changes are sent for changes made after the database was opened. KCalCore is based on the concept of loading a calendar (partially) into the client, modifying it there, then writing it back. Changes are also sent while the storage is open.

TODO:

=== Suppress Unnecessary Data Transfer ===

EDS currently sends all data when a view is requested, which is unnecessary when the client is only interested in change notifications. An extension to the EDS query language could optionally suppresses that sending of existing data when the view is opened, to be implemented only for the local contacts and calendar backends. This would allow easier integration with the QContact API.

The Fremantle EDS fork also had a “freezable” (static) capability for libebook, which should probably be investigated. See, for instance: http://maemo.org/api_refs/5.0/5.0-final/libebook/EBookView.html#e-book-view-set-freezable

One possibility for adding the flag is to (ab)use the existing query language with a match for special fields:
* in the client: query = e_book_query_orv(e_book_query_field_exists("X-EDS-QUERY-FLAG-SKIP-DATA-DUMP"), <real query>, NULL);
* in the server: "if (query is OR-Query and first clause is exists("X-EDS-QUERY-FLAG-SKIP-DATA-DUMP")) { skip_data_dump = TRUE; query = second clause; }

=== Return UIDs Only ===

The e_book_get_book_view() function takes a list of requested fields, to avoid returning unnecessary data, but this is apparently ignored for the local contacts backend. It would anyway require extra parsing and generation of smaller VCards, losing some possible advantages from delayed parsing of VCards.

However, the UID is already stored separately for use as the primary key of the BDB database, so it would be worthwhile to support that particular requested field. This would allow easier integration with the QContact API. It would also offer much better performance in cases where the client already has the data but just wants to discover what subset of that data meets a certain criteria.

Chris discovered a dedicated API for this in the Fremantle EDS fork. The new API call was accepted upstream and [https://bugzilla.gnome.org/show_bug.cgi?id=651446#c11 backported as a patch] (used in the Meego package) for the gnome-2-32, so no further work is needed?

== Optional Parsing of ebook Data in Client ==

The EDS client-side APIs currently parse and (temporarily) store the parsed details of VCards. However, this is often unnecessary because APIs such as e_book_get_book_view() cause it to return the whole VCard anyway. There will be some performance gain from delaying VCard parsing until it is really needed by calls to functions such as e_contact_new_from_vcard().

libebook provides different functionality:
# exchange data as vCard with EDS daemon
# parse items and wrap them in a C API

QtContacts replaces the second part. An obvious performance improvement is to only do the second part if the additional C APIs are really called. The following calls then transfer data without any additional parsing in libebook:
* e_book_get_contact()/asynchronous read/view + e_vcard_to_string()
* e_contact_new_from_vcard() + e_book_add_contact()/e_book_commit_contact()/asynchronous storing

Note that libecal (for iCalendar) can already do this, though libebook can't yet.

TODO:
# add delayed parsing to libebook, triggered by API call which needs parsed data

PROBLEM:
# If delayed parsing fails, how can error be reported?
# Should delayed parsing be done always (e_vcard_to_string())) or only if requested? How would that be configured by a libebook user?

== Asynchronous API for storing items in libecal ==

libecal should allow the caller to request storing of data without blocking on processing of the data. It should notify the caller when the data has been stored, or when processing has failed.

These functions may require async versions:
* icalcomponent_new_from_string()
* e_cal_modify_object[_with_mod]
* e_cal_create_object()

== More Efficient Access to Meta Data for Change Tracking ==

This was discussed a while ago: [[http://www.mail-archive.com/evolution-hackers@gnome.org/msg03029.html
[Evolution-hackers] concurrent modifications of items in GUI and EDS database]]

Data synchronization only needs to know which items exist (list of local IDs) and which revision of each item is currently stored. Revisions can be identified with a string which changes for each modification. Restoring an older revision should (but doesn't have to) restore the revision string.

For calendar, UID+RECURRENCE-ID and LAST-MODIFIED can be used. For contacts, UID + REV.

By comparing a stored list against the current one, changes can be detected without relying on journals in the daemon or comparison of time stamps. Journals are hard to maintain (how many of them, how long?). Comparison of time stamps has problems when changes are allowed while a sync runs and assumes that system time is linear (which is not always true).

As an additional use case, local ID + revision string could be used to prevent overwriting stale data in the server, as in HTTP PUT eTag checks.

TODO:

The Maemo Fremantle port of EDS, which is of course already open source, apparently has some solution for this, according to Mathias Hasselman, so that should be investigated first.

However, this should probably be made possible by:

# Extending the EDS query language with a flag that limits the delivered data to just:
## UID and REV for libebook (REV is in VCard 3.0)
## UID+RECURRENCE-ID+LAST-MODIFIED, for libecal
# Implementing these queries for the local contact and calendar backends. The query would be rejected, with an error, for other backends.

== Atomic Updates ==

Later, implement atomic changes as discussed in the mail thread: [[http://www.mail-archive.com/evolution-hackers@gnome.org/msg03029.html

== Quick check for "data changed" ==

The libebook and libical APIs should offer a way to discover if any data whatsoever has changed. This would, for instance, help SyncEvolution to avoid unnecessary detailed checking. This could just check the modification dates of local storage files, such as the BDB .db file, or .ics file, as long as we do not use those files to store permanent caches of parsed data. This would be much quicker than listing items for the whole database and comparing their revisions strings, as is currently necessary.

This would require small API additions, to check that the opaque revision strings are unchanged, such as

gchar* e_book_get_revision(EBook *book)
gboolean e_book_check_revision(EBook *book , const gchar* revision)

and

gchar* e_cal_get_revision(ECal *ecal)
gboolean e_cal_check_revision(ECal ecal, const gchar revision)

The semantic would be this:
* If the revision string is empty or different, then the content of the database '''might''' have changed. A more detailed check is necessary to determine if it has really changed. This will be the case for non-local storage.
* If it is non-empty and the same as before, the nothing has changed. No further check will be necessary.

== Contacts: Store PHOTO Data as Plain Files ==

Photos should be transferred over D-Bus via a filepath to a local file rather than transferring the actual photo data over D-Bus. The VCard format already allows this option (See the PHOTO property). We must be careful to keep these local files in sync and to remove them when contacts are removed, and to restrict access to the files from other processes.

There should be utility code available to libebook users to convert between the different representations.

Goals:
# Apps should be able to create photo files without ever passing the data to libebook.
# External PHOTO file and corresponding contact must be kept in sync:
** If a contact is deleted, the photo also needs to be removed.
** If a photo file is created and then storing the contact fails, who removes the file?
# Access to photo data must be limited to processes which have the right permissions.

Proposal:
* For each ~/.local/share/evolution/foo calendar, store photos in <photo path>=~/.local/share/evolution/foo/photos, with same permissions.
* New photo files are created with a [http://en.wikipedia.org/wiki/Universally_unique_identifier UUID] as base name and suffix according to the file content.
* If a contact is deleted and has a PHOTO;VALUE=uri:file://<photo path>/*, delete that file.
* For apps which want to create the photo file directly:
** Apps get a file descriptor and the path from a new libebook API.
** They own the file until the EDS server acknowledges the successful storing of a contact with a matching PHOTO;VALUE=uri:file property. If that fails, the app must remove the file.
* Add new API calls to libebook which split out resp. include data inline.
** Must track ownership of file after splitting it out and before storing - might never get stored!

Open:
* Should there be automatic garbage collection of stale photos in case something went wrong and a photo wasn't properly deleted?
* Trust TYPE=JPEG/PNG/... or look into file content to determine type?

== Future: Content Protection ==

Only authorized applications (= processes) with the necessary permissions should be able to read/write data. Exact permissions and mechanisms for verifying them to be defined as part of the (revised?) MeeGo architecture.

This can be added as follows:
# lock down file permissions
# add permission checks to EDS D-Bus servers

Currently not planned. Depends on the future MeeGo security infrastructure.

Architecture/planning/evolution-data-server/eds-improvements

2011-06-08T11:57:01Z

Murrayc:

This page describes improvements that are desirable or even essential for good performance of Evolution Data Server (EDS) as PIM storage in MeeGo.

All of these changes should be developed for the EDS master branch, reviewed by upstream maintainers and then back-ported to EDS 2.32, the version in MeeGo.

== Change Tracking ==

EDS is based on the model/view/controller principle: an application opens a database (calendar or addressbook) and requests a view with server-side filtering, then the application is sent:
# all data covered by the view
# the "all data sent" signal
# changes (add/update/remove) while the view exists

Applications modify data with asynchronous requests (controller) and keep showing the old data until the change notification comes in. The exact changes made to the data are determined by the server (for example when meeting invitations are sent).

QtContacts and KCalCore do not have such a strict model. With QtContacts the app simply loads and stores contacts and is responsible for maintaining its own view. Changes are sent for changes made after the database was opened. KCalCore is based on the concept of loading a calendar (partially) into the client, modifying it there, then writing it back. Changes are also sent while the storage is open.

TODO:

=== Suppress Unnecessary Data Transfer ===

EDS currently sends all data when a view is requested, which is unnecessary when the client is only interested in change notifications. An extension to the EDS query language could optionally suppresses that sending of existing data when the view is opened, to be implemented only for the local contacts and calendar backends. This would allow easier integration with the QContact API.

The Fremantle EDS fork also had a “freezable” (static) capability for libebook, which should probably be investigated. See, for instance: http://maemo.org/api_refs/5.0/5.0-final/libebook/EBookView.html#e-book-view-set-freezable

One possibility for adding the flag is to (ab)use the existing query language with a match for special fields:
* in the client: query = e_book_query_orv(e_book_query_field_exists("X-EDS-QUERY-FLAG-SKIP-DATA-DUMP"), <real query>, NULL);
* in the server: "if (query is OR-Query and first clause is exists("X-EDS-QUERY-FLAG-SKIP-DATA-DUMP")) { skip_data_dump = TRUE; query = second clause; }

=== Return UIDs Only ===

The e_book_get_book_view() function takes a list of requested fields, to avoid returning unnecessary data, but this is apparently ignored for the local contacts backend. It would anyway require extra parsing and generation of smaller VCards, losing some possible advantages from delayed parsing of VCards.

However, the UID is already stored separately for use as the primary key of the BDB database, so it would be worthwhile to support that particular requested field. This would allow easier integration with the QContact API. It would also offer much better performance in cases where the client already has the data but just wants to discover what subset of that data meets a certain criteria.

Chris discovered a dedicated API for this in the Fremantle EDS fork. The new API call was accepted upstream and backported to gnome-2-32, so no further work is needed?

== Optional Parsing of ebook Data in Client ==

The EDS client-side APIs currently parse and (temporarily) store the parsed details of VCards. However, this is often unnecessary because APIs such as e_book_get_book_view() cause it to return the whole VCard anyway. There will be some performance gain from delaying VCard parsing until it is really needed by calls to functions such as e_contact_new_from_vcard().

libebook provides different functionality:
# exchange data as vCard with EDS daemon
# parse items and wrap them in a C API

QtContacts replaces the second part. An obvious performance improvement is to only do the second part if the additional C APIs are really called. The following calls then transfer data without any additional parsing in libebook:
* e_book_get_contact()/asynchronous read/view + e_vcard_to_string()
* e_contact_new_from_vcard() + e_book_add_contact()/e_book_commit_contact()/asynchronous storing

Note that libecal (for iCalendar) can already do this, though libebook can't yet.

TODO:
# add delayed parsing to libebook, triggered by API call which needs parsed data

PROBLEM:
# If delayed parsing fails, how can error be reported?
# Should delayed parsing be done always (e_vcard_to_string())) or only if requested? How would that be configured by a libebook user?

== Asynchronous API for storing items in libecal ==

libecal should allow the caller to request storing of data without blocking on processing of the data. It should notify the caller when the data has been stored, or when processing has failed.

These functions may require async versions:
* icalcomponent_new_from_string()
* e_cal_modify_object[_with_mod]
* e_cal_create_object()

== More Efficient Access to Meta Data for Change Tracking ==

This was discussed a while ago: [[http://www.mail-archive.com/evolution-hackers@gnome.org/msg03029.html
[Evolution-hackers] concurrent modifications of items in GUI and EDS database]]

Data synchronization only needs to know which items exist (list of local IDs) and which revision of each item is currently stored. Revisions can be identified with a string which changes for each modification. Restoring an older revision should (but doesn't have to) restore the revision string.

For calendar, UID+RECURRENCE-ID and LAST-MODIFIED can be used. For contacts, UID + REV.

By comparing a stored list against the current one, changes can be detected without relying on journals in the daemon or comparison of time stamps. Journals are hard to maintain (how many of them, how long?). Comparison of time stamps has problems when changes are allowed while a sync runs and assumes that system time is linear (which is not always true).

As an additional use case, local ID + revision string could be used to prevent overwriting stale data in the server, as in HTTP PUT eTag checks.

TODO:

The Maemo Fremantle port of EDS, which is of course already open source, apparently has some solution for this, according to Mathias Hasselman, so that should be investigated first.

However, this should probably be made possible by:

# Extending the EDS query language with a flag that limits the delivered data to just:
## UID and REV for libebook (REV is in VCard 3.0)
## UID+RECURRENCE-ID+LAST-MODIFIED, for libecal
# Implementing these queries for the local contact and calendar backends. The query would be rejected, with an error, for other backends.

== Atomic Updates ==

Later, implement atomic changes as discussed in the mail thread: [[http://www.mail-archive.com/evolution-hackers@gnome.org/msg03029.html

== Quick check for "data changed" ==

The libebook and libical APIs should offer a way to discover if any data whatsoever has changed. This would, for instance, help SyncEvolution to avoid unnecessary detailed checking. This could just check the modification dates of local storage files, such as the BDB .db file, or .ics file, as long as we do not use those files to store permanent caches of parsed data. This would be much quicker than listing items for the whole database and comparing their revisions strings, as is currently necessary.

This would require small API additions, to check that the opaque revision strings are unchanged, such as

gchar* e_book_get_revision(EBook *book)
gboolean e_book_check_revision(EBook *book , const gchar* revision)

and

gchar* e_cal_get_revision(ECal *ecal)
gboolean e_cal_check_revision(ECal ecal, const gchar revision)

The semantic would be this:
* If the revision string is empty or different, then the content of the database '''might''' have changed. A more detailed check is necessary to determine if it has really changed. This will be the case for non-local storage.
* If it is non-empty and the same as before, the nothing has changed. No further check will be necessary.

== Contacts: Store PHOTO Data as Plain Files ==

Photos should be transferred over D-Bus via a filepath to a local file rather than transferring the actual photo data over D-Bus. The VCard format already allows this option (See the PHOTO property). We must be careful to keep these local files in sync and to remove them when contacts are removed, and to restrict access to the files from other processes.

There should be utility code available to libebook users to convert between the different representations.

Goals:
# Apps should be able to create photo files without ever passing the data to libebook.
# External PHOTO file and corresponding contact must be kept in sync:
** If a contact is deleted, the photo also needs to be removed.
** If a photo file is created and then storing the contact fails, who removes the file?
# Access to photo data must be limited to processes which have the right permissions.

Proposal:
* For each ~/.local/share/evolution/foo calendar, store photos in <photo path>=~/.local/share/evolution/foo/photos, with same permissions.
* New photo files are created with a [http://en.wikipedia.org/wiki/Universally_unique_identifier UUID] as base name and suffix according to the file content.
* If a contact is deleted and has a PHOTO;VALUE=uri:file://<photo path>/*, delete that file.
* For apps which want to create the photo file directly:
** Apps get a file descriptor and the path from a new libebook API.
** They own the file until the EDS server acknowledges the successful storing of a contact with a matching PHOTO;VALUE=uri:file property. If that fails, the app must remove the file.
* Add new API calls to libebook which split out resp. include data inline.
** Must track ownership of file after splitting it out and before storing - might never get stored!

Open:
* Should there be automatic garbage collection of stale photos in case something went wrong and a photo wasn't properly deleted?
* Trust TYPE=JPEG/PNG/... or look into file content to determine type?

== Future: Content Protection ==

Only authorized applications (= processes) with the necessary permissions should be able to read/write data. Exact permissions and mechanisms for verifying them to be defined as part of the (revised?) MeeGo architecture.

This can be added as follows:
# lock down file permissions
# add permission checks to EDS D-Bus servers

Currently not planned. Depends on the future MeeGo security infrastructure.

Architecture/planning/syncevolution/syncevolution-improvements

2011-06-08T11:38:04Z

Murrayc: Created page with "= SyncEvolution Improvements = * [https://bugs.meego.com/show_bug.cgi?id=736#c2 Detect Manufacturer/Model via Bluetooth] * [https://bugs.meego.com/show_bug.cgi?id=1351#c1 Initi..."

= SyncEvolution Improvements =

* [https://bugs.meego.com/show_bug.cgi?id=736#c2 Detect Manufacturer/Model via Bluetooth]
* [https://bugs.meego.com/show_bug.cgi?id=1351#c1 Initial refactoring: Reorganization of code]
* [https://bugs.meego.com/show_bug.cgi?id=716#c2 Refactor code to use forked processes]
* [https://bugs.meego.com/show_bug.cgi?id=1351 Syncevo-dbus-server Clean Up (Complete rewrite to deal with arising issues)]

Architecture/planning/evolution-data-server/eds-improvements

2011-06-08T11:29:53Z

Murrayc: Move Future stuff to the bottom.

This page describes improvements that are desirable or even essential for good performance of Evolution Data Server (EDS) as PIM storage in MeeGo.

All of these changes should be developed for the EDS master branch, reviewed by upstream maintainers and then back-ported to EDS 2.32, the version in MeeGo.

== Change Tracking ==

EDS is based on the model/view/controller principle: an application opens a database (calendar or addressbook) and requests a view with server-side filtering, then the application is sent:
# all data covered by the view
# the "all data sent" signal
# changes (add/update/remove) while the view exists

Applications modify data with asynchronous requests (controller) and keep showing the old data until the change notification comes in. The exact changes made to the data are determined by the server (for example when meeting invitations are sent).

QtContacts and KCalCore do not have such a strict model. With QtContacts the app simply loads and stores contacts and is responsible for maintaining its own view. Changes are sent for changes made after the database was opened. KCalCore is based on the concept of loading a calendar (partially) into the client, modifying it there, then writing it back. Changes are also sent while the storage is open.

TODO:

=== Suppress Unnecessary Data Transfer ===

EDS currently sends all data when a view is requested, which is unnecessary when the client is only interested in change notifications. An extension to the EDS query language could optionally suppresses that sending of existing data when the view is opened, to be implemented only for the local contacts and calendar backends. This would allow easier integration with the QContact API.

The Fremantle EDS fork also had a “freezable” (static) capability for libebook, which should probably be investigated. See, for instance: http://maemo.org/api_refs/5.0/5.0-final/libebook/EBookView.html#e-book-view-set-freezable

One possibility for adding the flag is to (ab)use the existing query language with a match for special fields:
* in the client: query = e_book_query_orv(e_book_query_field_exists("X-EDS-QUERY-FLAG-SKIP-DATA-DUMP"), <real query>, NULL);
* in the server: "if (query is OR-Query and first clause is exists("X-EDS-QUERY-FLAG-SKIP-DATA-DUMP")) { skip_data_dump = TRUE; query = second clause; }

=== Return UIDs Only ===

The e_book_get_book_view() function takes a list of requested fields, to avoid returning unnecessary data, but this is apparently ignored for the local contacts backend. It would anyway require extra parsing and generation of smaller VCards, losing some possible advantages from delayed parsing of VCards.

However, the UID is already stored separately for use as the primary key of the BDB database, so it would be worthwhile to support that particular requested field. This would allow easier integration with the QContact API. It would also offer much better performance in cases where the client already has the data but just wants to discover what subset of that data meets a certain criteria.

Chris discovered a dedicated API for this in the Fremantle EDS fork. The new API call was accepted upstream and backported to gnome-2-32, so no further work is needed?

== Optional Parsing of ebook Data in Client ==

The EDS client-side APIs currently parse and (temporarily) store the parsed details of VCards. However, this is often unnecessary because APIs such as e_book_get_book_view() cause it to return the whole VCard anyway. There will be some performance gain from delaying VCard parsing until it is really needed by calls to functions such as e_contact_new_from_vcard().

libebook provides different functionality:
# exchange data as vCard with EDS daemon
# parse items and wrap them in a C API

QtContacts replaces the second part. An obvious performance improvement is to only do the second part if the additional C APIs are really called. The following calls then transfer data without any additional parsing in libebook:
* e_book_get_contact()/asynchronous read/view + e_vcard_to_string()
* e_contact_new_from_vcard() + e_book_add_contact()/e_book_commit_contact()/asynchronous storing

Note that libecal (for iCalendar) can already do this, though libebook can't yet.

TODO:
# add delayed parsing to libebook, triggered by API call which needs parsed data

PROBLEM:
# If delayed parsing fails, how can error be reported?
# Should delayed parsing be done always (e_vcard_to_string())) or only if requested? How would that be configured by a libebook user?

== More Efficient Access to Meta Data for Change Tracking ==

This was discussed a while ago: [[http://www.mail-archive.com/evolution-hackers@gnome.org/msg03029.html
[Evolution-hackers] concurrent modifications of items in GUI and EDS database]]

Data synchronization only needs to know which items exist (list of local IDs) and which revision of each item is currently stored. Revisions can be identified with a string which changes for each modification. Restoring an older revision should (but doesn't have to) restore the revision string.

For calendar, UID+RECURRENCE-ID and LAST-MODIFIED can be used. For contacts, UID + REV.

By comparing a stored list against the current one, changes can be detected without relying on journals in the daemon or comparison of time stamps. Journals are hard to maintain (how many of them, how long?). Comparison of time stamps has problems when changes are allowed while a sync runs and assumes that system time is linear (which is not always true).

As an additional use case, local ID + revision string could be used to prevent overwriting stale data in the server, as in HTTP PUT eTag checks.

TODO:

The Maemo Fremantle port of EDS, which is of course already open source, apparently has some solution for this, according to Mathias Hasselman, so that should be investigated first.

However, this should probably be made possible by:

# Extending the EDS query language with a flag that limits the delivered data to just:
## UID and REV for libebook (REV is in VCard 3.0)
## UID+RECURRENCE-ID+LAST-MODIFIED, for libecal
# Implementing these queries for the local contact and calendar backends. The query would be rejected, with an error, for other backends.

== Atomic Updates ==

Later, implement atomic changes as discussed in the mail thread: [[http://www.mail-archive.com/evolution-hackers@gnome.org/msg03029.html

== Quick check for "data changed" ==

The libebook and libical APIs should offer a way to discover if any data whatsoever has changed. This would, for instance, help SyncEvolution to avoid unnecessary detailed checking. This could just check the modification dates of local storage files, such as the BDB .db file, or .ics file, as long as we do not use those files to store permanent caches of parsed data. This would be much quicker than listing items for the whole database and comparing their revisions strings, as is currently necessary.

This would require small API additions, to check that the opaque revision strings are unchanged, such as

gchar* e_book_get_revision(EBook *book)
gboolean e_book_check_revision(EBook *book , const gchar* revision)

and

gchar* e_cal_get_revision(ECal *ecal)
gboolean e_cal_check_revision(ECal ecal, const gchar revision)

The semantic would be this:
* If the revision string is empty or different, then the content of the database '''might''' have changed. A more detailed check is necessary to determine if it has really changed. This will be the case for non-local storage.
* If it is non-empty and the same as before, the nothing has changed. No further check will be necessary.

== Contacts: Store PHOTO Data as Plain Files ==

Photos should be transferred over D-Bus via a filepath to a local file rather than transferring the actual photo data over D-Bus. The VCard format already allows this option (See the PHOTO property). We must be careful to keep these local files in sync and to remove them when contacts are removed, and to restrict access to the files from other processes.

There should be utility code available to libebook users to convert between the different representations.

Goals:
# Apps should be able to create photo files without ever passing the data to libebook.
# External PHOTO file and corresponding contact must be kept in sync:
** If a contact is deleted, the photo also needs to be removed.
** If a photo file is created and then storing the contact fails, who removes the file?
# Access to photo data must be limited to processes which have the right permissions.

Proposal:
* For each ~/.local/share/evolution/foo calendar, store photos in <photo path>=~/.local/share/evolution/foo/photos, with same permissions.
* New photo files are created with a [http://en.wikipedia.org/wiki/Universally_unique_identifier UUID] as base name and suffix according to the file content.
* If a contact is deleted and has a PHOTO;VALUE=uri:file://<photo path>/*, delete that file.
* For apps which want to create the photo file directly:
** Apps get a file descriptor and the path from a new libebook API.
** They own the file until the EDS server acknowledges the successful storing of a contact with a matching PHOTO;VALUE=uri:file property. If that fails, the app must remove the file.
* Add new API calls to libebook which split out resp. include data inline.
** Must track ownership of file after splitting it out and before storing - might never get stored!

Open:
* Should there be automatic garbage collection of stale photos in case something went wrong and a photo wasn't properly deleted?
* Trust TYPE=JPEG/PNG/... or look into file content to determine type?

== Future: Content Protection ==

Only authorized applications (= processes) with the necessary permissions should be able to read/write data. Exact permissions and mechanisms for verifying them to be defined as part of the (revised?) MeeGo architecture.

This can be added as follows:
# lock down file permissions
# add permission checks to EDS D-Bus servers

Currently not planned. Depends on the future MeeGo security infrastructure.

Architecture/planning/evolution-data-server/eds-improvements

2011-06-08T11:23:21Z

Murrayc: Optional Parsing of ebook Data in Client: Add some explanatory text.

This page describes improvements that are desirable or even essential for good performance of Evolution Data Server (EDS) as PIM storage in MeeGo.

All of these changes should be developed for the EDS master branch, reviewed by upstream maintainers and then back-ported to EDS 2.32, the version in MeeGo.

== Future: Content Protection ==

Only authorized applications (= processes) with the necessary permissions should be able to read/write data. Exact permissions and mechanisms for verifying them to be defined as part of the (revised?) MeeGo architecture.

This can be added as follows:
# lock down file permissions
# add permission checks to EDS D-Bus servers

Currently not planned. Depends on the future MeeGo security infrastructure.

== Change Tracking ==

EDS is based on the model/view/controller principle: an application opens a database (calendar or addressbook) and requests a view with server-side filtering, then the application is sent:
# all data covered by the view
# the "all data sent" signal
# changes (add/update/remove) while the view exists

Applications modify data with asynchronous requests (controller) and keep showing the old data until the change notification comes in. The exact changes made to the data are determined by the server (for example when meeting invitations are sent).

QtContacts and KCalCore do not have such a strict model. With QtContacts the app simply loads and stores contacts and is responsible for maintaining its own view. Changes are sent for changes made after the database was opened. KCalCore is based on the concept of loading a calendar (partially) into the client, modifying it there, then writing it back. Changes are also sent while the storage is open.

TODO:

=== Suppress Unnecessary Data Transfer ===

EDS currently sends all data when a view is requested, which is unnecessary when the client is only interested in change notifications. An extension to the EDS query language could optionally suppresses that sending of existing data when the view is opened, to be implemented only for the local contacts and calendar backends. This would allow easier integration with the QContact API.

The Fremantle EDS fork also had a “freezable” (static) capability for libebook, which should probably be investigated. See, for instance: http://maemo.org/api_refs/5.0/5.0-final/libebook/EBookView.html#e-book-view-set-freezable

One possibility for adding the flag is to (ab)use the existing query language with a match for special fields:
* in the client: query = e_book_query_orv(e_book_query_field_exists("X-EDS-QUERY-FLAG-SKIP-DATA-DUMP"), <real query>, NULL);
* in the server: "if (query is OR-Query and first clause is exists("X-EDS-QUERY-FLAG-SKIP-DATA-DUMP")) { skip_data_dump = TRUE; query = second clause; }

=== Return UIDs Only ===

The e_book_get_book_view() function takes a list of requested fields, to avoid returning unnecessary data, but this is apparently ignored for the local contacts backend. It would anyway require extra parsing and generation of smaller VCards, losing some possible advantages from delayed parsing of VCards.

However, the UID is already stored separately for use as the primary key of the BDB database, so it would be worthwhile to support that particular requested field. This would allow easier integration with the QContact API. It would also offer much better performance in cases where the client already has the data but just wants to discover what subset of that data meets a certain criteria.

Chris discovered a dedicated API for this in the Fremantle EDS fork. The new API call was accepted upstream and backported to gnome-2-32, so no further work is needed?

== Optional Parsing of ebook Data in Client ==

The EDS client-side APIs currently parse and (temporarily) store the parsed details of VCards. However, this is often unnecessary because APIs such as e_book_get_book_view() cause it to return the whole VCard anyway. There will be some performance gain from delaying VCard parsing until it is really needed by calls to functions such as e_contact_new_from_vcard().

libebook provides different functionality:
# exchange data as vCard with EDS daemon
# parse items and wrap them in a C API

QtContacts replaces the second part. An obvious performance improvement is to only do the second part if the additional C APIs are really called. The following calls then transfer data without any additional parsing in libebook:
* e_book_get_contact()/asynchronous read/view + e_vcard_to_string()
* e_contact_new_from_vcard() + e_book_add_contact()/e_book_commit_contact()/asynchronous storing

Note that libecal (for iCalendar) can already do this, though libebook can't yet.

TODO:
# add delayed parsing to libebook, triggered by API call which needs parsed data

PROBLEM:
# If delayed parsing fails, how can error be reported?
# Should delayed parsing be done always (e_vcard_to_string())) or only if requested? How would that be configured by a libebook user?

== More Efficient Access to Meta Data for Change Tracking ==

This was discussed a while ago: [[http://www.mail-archive.com/evolution-hackers@gnome.org/msg03029.html
[Evolution-hackers] concurrent modifications of items in GUI and EDS database]]

Data synchronization only needs to know which items exist (list of local IDs) and which revision of each item is currently stored. Revisions can be identified with a string which changes for each modification. Restoring an older revision should (but doesn't have to) restore the revision string.

For calendar, UID+RECURRENCE-ID and LAST-MODIFIED can be used. For contacts, UID + REV.

By comparing a stored list against the current one, changes can be detected without relying on journals in the daemon or comparison of time stamps. Journals are hard to maintain (how many of them, how long?). Comparison of time stamps has problems when changes are allowed while a sync runs and assumes that system time is linear (which is not always true).

As an additional use case, local ID + revision string could be used to prevent overwriting stale data in the server, as in HTTP PUT eTag checks.

TODO:

The Maemo Fremantle port of EDS, which is of course already open source, apparently has some solution for this, according to Mathias Hasselman, so that should be investigated first.

However, this should probably be made possible by:

# Extending the EDS query language with a flag that limits the delivered data to just:
## UID and REV for libebook (REV is in VCard 3.0)
## UID+RECURRENCE-ID+LAST-MODIFIED, for libecal
# Implementing these queries for the local contact and calendar backends. The query would be rejected, with an error, for other backends.

== Atomic Updates ==

Later, implement atomic changes as discussed in the mail thread: [[http://www.mail-archive.com/evolution-hackers@gnome.org/msg03029.html

== Quick check for "data changed" ==

The libebook and libical APIs should offer a way to discover if any data whatsoever has changed. This would, for instance, help SyncEvolution to avoid unnecessary detailed checking. This could just check the modification dates of local storage files, such as the BDB .db file, or .ics file, as long as we do not use those files to store permanent caches of parsed data. This would be much quicker than listing items for the whole database and comparing their revisions strings, as is currently necessary.

This would require small API additions, to check that the opaque revision strings are unchanged, such as

gchar* e_book_get_revision(EBook *book)
gboolean e_book_check_revision(EBook *book , const gchar* revision)

and

gchar* e_cal_get_revision(ECal *ecal)
gboolean e_cal_check_revision(ECal ecal, const gchar revision)

The semantic would be this:
* If the revision string is empty or different, then the content of the database '''might''' have changed. A more detailed check is necessary to determine if it has really changed. This will be the case for non-local storage.
* If it is non-empty and the same as before, the nothing has changed. No further check will be necessary.

== Contacts: Store PHOTO Data as Plain Files ==

Photos should be transferred over D-Bus via a filepath to a local file rather than transferring the actual photo data over D-Bus. The VCard format already allows this option (See the PHOTO property). We must be careful to keep these local files in sync and to remove them when contacts are removed, and to restrict access to the files from other processes.

There should be utility code available to libebook users to convert between the different representations.

Goals:
# Apps should be able to create photo files without ever passing the data to libebook.
# External PHOTO file and corresponding contact must be kept in sync:
** If a contact is deleted, the photo also needs to be removed.
** If a photo file is created and then storing the contact fails, who removes the file?
# Access to photo data must be limited to processes which have the right permissions.

Proposal:
* For each ~/.local/share/evolution/foo calendar, store photos in <photo path>=~/.local/share/evolution/foo/photos, with same permissions.
* New photo files are created with a [http://en.wikipedia.org/wiki/Universally_unique_identifier UUID] as base name and suffix according to the file content.
* If a contact is deleted and has a PHOTO;VALUE=uri:file://<photo path>/*, delete that file.
* For apps which want to create the photo file directly:
** Apps get a file descriptor and the path from a new libebook API.
** They own the file until the EDS server acknowledges the successful storing of a contact with a matching PHOTO;VALUE=uri:file property. If that fails, the app must remove the file.
* Add new API calls to libebook which split out resp. include data inline.
** Must track ownership of file after splitting it out and before storing - might never get stored!

Open:
* Should there be automatic garbage collection of stale photos in case something went wrong and a photo wasn't properly deleted?
* Trust TYPE=JPEG/PNG/... or look into file content to determine type?

Architecture/planning/evolution-data-server/eds-improvements

2011-06-08T10:11:40Z

Murrayc: Title capitalization

This page describes improvements that are desirable or even essential for good performance of Evolution Data Server (EDS) as PIM storage in MeeGo.

All of these changes should be developed for the EDS master branch, reviewed by upstream maintainers and then back-ported to EDS 2.32, the version in MeeGo.

== Content Protection ==

Only authorized applications (= processes) with the necessary permissions should be able to read/write data. Exact permissions and mechanisms for verifying them to be defined as part of the (revised?) MeeGo architecture.

This can be added as follows:
# lock down file permissions
# add permission checks to EDS D-Bus servers

== Change Tracking ==

EDS is based on the model/view/controller principle: an application opens a database (calendar or addressbook) and requests a view with server-side filtering, then the application is sent:
# all data covered by the view
# the "all data sent" signal
# changes (add/update/remove) while the view exists

Applications modify data with asynchronous requests (controller) and keep showing the old data until the change notification comes in. The exact changes made to the data are determined by the server (for example when meeting invitations are sent).

QtContacts and KCalCore do not have such a strict model. With QtContacts the app simply loads and stores contacts and is responsible for maintaining its own view. Changes are sent for changes made after the database was opened. KCalCore is based on the concept of loading a calendar (partially) into the client, modifying it there, then writing it back. Changes are also sent while the storage is open.

TODO:

=== Suppress Unnecessary Data Transfer ===

EDS currently sends all data when a view is requested, which is unnecessary when the client is only interested in change notifications. An extension to the EDS query language could optionally suppresses that sending of existing data when the view is opened, to be implemented only for the local contacts and calendar backends. This would allow easier integration with the QContact API.

The Fremantle EDS fork also had a “freezable” (static) capability for libebook, which should probably be investigated. See, for instance: http://maemo.org/api_refs/5.0/5.0-final/libebook/EBookView.html#e-book-view-set-freezable

One possibility for adding the flag is to (ab)use the existing query language with a match for special fields:
* in the client: query = e_book_query_orv(e_book_query_field_exists("X-EDS-QUERY-FLAG-SKIP-DATA-DUMP"), <real query>, NULL);
* in the server: "if (query is OR-Query and first clause is exists("X-EDS-QUERY-FLAG-SKIP-DATA-DUMP")) { skip_data_dump = TRUE; query = second clause; }

=== Return UIDs Only ===

The e_book_get_book_view() function takes a list of requested fields, to avoid returning unnecessary data, but this is apparently ignored for the local contacts backend. It would anyway require extra parsing and generation of smaller VCards, losing some possible advantages from delayed parsing of VCards.

However, the UID is already stored separately for use as the primary key of the BDB database, so it would be worthwhile to support that particular requested field. This would allow easier integration with the QContact API. It would also offer much better performance in cases where the client already has the data but just wants to discover what subset of that data meets a certain criteria.

== Optional Parsing of ebook Data in Client ==

libebook provides different functionality:
# exchange data as vCard with EDS daemon
# parse items and wrap them in a C API

QtContacts replaces the second part. An obvious performance improvement is to only do the second part if the additional C APIs are really called. The following calls then transfer data without any additional parsing in libebook:
* e_book_get_contact()/asynchronous read/view + e_vcard_to_string()
* e_contact_new_from_vcard() + e_book_add_contact()/e_book_commit_contact()/asynchronous storing

Note that libecal (for iCalendar) can already do this, though libebook can't yet.

TODO:
# add delayed parsing to libebook, triggered by API call which needs parsed data

PROBLEM:
# If delayed parsing fails, how can error be reported?
# Should delayed parsing be done always (e_vcard_to_string())) or only if requested? How would that be configured by a libebook user?

== More Efficient Access to Meta Data for Change Tracking ==

This was discussed a while ago: [[http://www.mail-archive.com/evolution-hackers@gnome.org/msg03029.html
[Evolution-hackers] concurrent modifications of items in GUI and EDS database]]

Data synchronization only needs to know which items exist (list of local IDs) and which revision of each item is currently stored. Revisions can be identified with a string which changes for each modification. Restoring an older revision should (but doesn't have to) restore the revision string.

For calendar, UID+RECURRENCE-ID and LAST-MODIFIED can be used. For contacts, UID + REV.

By comparing a stored list against the current one, changes can be detected without relying on journals in the daemon or comparison of time stamps. Journals are hard to maintain (how many of them, how long?). Comparison of time stamps has problems when changes are allowed while a sync runs and assumes that system time is linear (which is not always true).

As an additional use case, local ID + revision string could be used to prevent overwriting stale data in the server, as in HTTP PUT eTag checks.

TODO:

The Maemo Fremantle port of EDS, which is of course already open source, apparently has some solution for this, according to Mathias Hasselman, so that should be investigated first.

However, this should probably be made possible by:

# Extending the EDS query language with a flag that limits the delivered data to just:
## UID and REV for libebook (REV is in VCard 3.0)
## UID+RECURRENCE-ID+LAST-MODIFIED, for libecal
# Implementing these queries for the local contact and calendar backends. The query would be rejected, with an error, for other backends.

== Atomic Updates ==

Later, implement atomic changes as discussed in the mail thread: [[http://www.mail-archive.com/evolution-hackers@gnome.org/msg03029.html

== Quick check for "data changed" ==

The libebook and libical APIs should offer a way to discover if any data whatsoever has changed. This would, for instance, help SyncEvolution to avoid unnecessary detailed checking. This could just check the modification dates of local storage files, such as the BDB .db file, or .ics file, as long as we do not use those files to store permanent caches of parsed data. This would be much quicker than listing items for the whole database and comparing their revisions strings, as is currently necessary.

This would require small API additions, to check that the opaque revision strings are unchanged, such as

gchar* e_book_get_revision(EBook *book)
gboolean e_book_check_revision(EBook *book , const gchar* revision)

and

gchar* e_cal_get_revision(ECal *ecal)
gboolean e_cal_check_revision(ECal ecal, const gchar revision)

The semantic would be this:
* If the revision string is empty or different, then the content of the database '''might''' have changed. A more detailed check is necessary to determine if it has really changed. This will be the case for non-local storage.
* If it is non-empty and the same as before, the nothing has changed. No further check will be necessary.

== Contacts: Store PHOTO Data as Plain Files ==

Photos should be transferred over D-Bus via a filepath to a local file rather than transferring the actual photo data over D-Bus. The VCard format already allows this option (See the PHOTO property). We must be careful to keep these local files in sync and to remove them when contacts are removed, and to restrict access to the files from other processes.

There should be utility code available to libebook users to convert between the different representations.

Goals:
# Apps should be able to create photo files without ever passing the data to libebook.
# External PHOTO file and corresponding contact must be kept in sync:
** If a contact is deleted, the photo also needs to be removed.
** If a photo file is created and then storing the contact fails, who removes the file?
# Access to photo data must be limited to processes which have the right permissions.

Proposal:
* For each ~/.local/share/evolution/foo calendar, store photos in <photo path>=~/.local/share/evolution/foo/photos, with same permissions.
* New photo files are created with a [http://en.wikipedia.org/wiki/Universally_unique_identifier UUID] as base name and suffix according to the file content.
* If a contact is deleted and has a PHOTO;VALUE=uri:file://<photo path>/*, delete that file.
* For apps which want to create the photo file directly:
** Apps get a file descriptor and the path from a new libebook API.
** They own the file until the EDS server acknowledges the successful storing of a contact with a matching PHOTO;VALUE=uri:file property. If that fails, the app must remove the file.
* Add new API calls to libebook which split out resp. include data inline.
** Must track ownership of file after splitting it out and before storing - might never get stored!

Open:
* Should there be automatic garbage collection of stale photos in case something went wrong and a photo wasn't properly deleted?
* Trust TYPE=JPEG/PNG/... or look into file content to determine type?

Architecture/planning/evolution-data-server/eds-improvements

2011-06-08T10:10:03Z

Murrayc: quick check for "data changed: clarification

This page describes improvements that are desirable or even essential for good performance of Evolution Data Server (EDS) as PIM storage in MeeGo.

All of these changes should be developed for the EDS master branch, reviewed by upstream maintainers and then back-ported to EDS 2.32, the version in MeeGo.

== content protection ==

Only authorized applications (= processes) with the necessary permissions should be able to read/write data. Exact permissions and mechanisms for verifying them to be defined as part of the (revised?) MeeGo architecture.

This can be added as follows:
# lock down file permissions
# add permission checks to EDS D-Bus servers

== change tracking ==

EDS is based on the model/view/controller principle: an application opens a database (calendar or addressbook) and requests a view with server-side filtering, then the application is sent:
# all data covered by the view
# the "all data sent" signal
# changes (add/update/remove) while the view exists

Applications modify data with asynchronous requests (controller) and keep showing the old data until the change notification comes in. The exact changes made to the data are determined by the server (for example when meeting invitations are sent).

QtContacts and KCalCore do not have such a strict model. With QtContacts the app simply loads and stores contacts and is responsible for maintaining its own view. Changes are sent for changes made after the database was opened. KCalCore is based on the concept of loading a calendar (partially) into the client, modifying it there, then writing it back. Changes are also sent while the storage is open.

TODO:

=== Suppress unnecessary data transfer ===

EDS currently sends all data when a view is requested, which is unnecessary when the client is only interested in change notifications. An extension to the EDS query language could optionally suppresses that sending of existing data when the view is opened, to be implemented only for the local contacts and calendar backends. This would allow easier integration with the QContact API.

The Fremantle EDS fork also had a “freezable” (static) capability for libebook, which should probably be investigated. See, for instance: http://maemo.org/api_refs/5.0/5.0-final/libebook/EBookView.html#e-book-view-set-freezable

One possibility for adding the flag is to (ab)use the existing query language with a match for special fields:
* in the client: query = e_book_query_orv(e_book_query_field_exists("X-EDS-QUERY-FLAG-SKIP-DATA-DUMP"), <real query>, NULL);
* in the server: "if (query is OR-Query and first clause is exists("X-EDS-QUERY-FLAG-SKIP-DATA-DUMP")) { skip_data_dump = TRUE; query = second clause; }

=== Return UIDs only ===

The e_book_get_book_view() function takes a list of requested fields, to avoid returning unnecessary data, but this is apparently ignored for the local contacts backend. It would anyway require extra parsing and generation of smaller VCards, losing some possible advantages from delayed parsing of VCards.

However, the UID is already stored separately for use as the primary key of the BDB database, so it would be worthwhile to support that particular requested field. This would allow easier integration with the QContact API. It would also offer much better performance in cases where the client already has the data but just wants to discover what subset of that data meets a certain criteria.

== optional parsing of ebook data in client ==

libebook provides different functionality:
# exchange data as vCard with EDS daemon
# parse items and wrap them in a C API

QtContacts replaces the second part. An obvious performance improvement is to only do the second part if the additional C APIs are really called. The following calls then transfer data without any additional parsing in libebook:
* e_book_get_contact()/asynchronous read/view + e_vcard_to_string()
* e_contact_new_from_vcard() + e_book_add_contact()/e_book_commit_contact()/asynchronous storing

Note that libecal (for iCalendar) can already do this, though libebook can't yet.

TODO:
# add delayed parsing to libebook, triggered by API call which needs parsed data

PROBLEM:
# If delayed parsing fails, how can error be reported?
# Should delayed parsing be done always (e_vcard_to_string())) or only if requested? How would that be configured by a libebook user?

== More efficient access to meta data for change tracking + atomic updates ==

This was discussed a while ago: [[http://www.mail-archive.com/evolution-hackers@gnome.org/msg03029.html
[Evolution-hackers] concurrent modifications of items in GUI and EDS database]]

Data synchronization only needs to know which items exist (list of local IDs) and which revision of each item is currently stored. Revisions can be identified with a string which changes for each modification. Restoring an older revision should (but doesn't have to) restore the revision string.

For calendar, UID+RECURRENCE-ID and LAST-MODIFIED can be used. For contacts, UID + REV.

By comparing a stored list against the current one, changes can be detected without relying on journals in the daemon or comparison of time stamps. Journals are hard to maintain (how many of them, how long?). Comparison of time stamps has problems when changes are allowed while a sync runs and assumes that system time is linear (which is not always true).

As an additional use case, local ID + revision string could be used to prevent overwriting stale data in the server, as in HTTP PUT eTag checks.

TODO:

The Maemo Fremantle port of EDS, which is of course already open source, apparently has some solution for this, according to Mathias Hasselman, so that should be investigated first.

However, this should probably be made possible by:

# Extending the EDS query language with a flag that limits the delivered data to just:
## UID and REV for libebook (REV is in VCard 3.0)
## UID+RECURRENCE-ID+LAST-MODIFIED, for libecal
# Implementing these queries for the local contact and calendar backends. The query would be rejected, with an error, for other backends.

== Atomic Updates ==

Later, implement atomic changes as discussed in the mail thread: [[http://www.mail-archive.com/evolution-hackers@gnome.org/msg03029.html

== Quick check for "data changed" ==

The libebook and libical APIs should offer a way to discover if any data whatsoever has changed. This would, for instance, help SyncEvolution to avoid unnecessary detailed checking. This could just check the modification dates of local storage files, such as the BDB .db file, or .ics file, as long as we do not use those files to store permanent caches of parsed data. This would be much quicker than listing items for the whole database and comparing their revisions strings, as is currently necessary.

This would require small API additions, to check that the opaque revision strings are unchanged, such as

gchar* e_book_get_revision(EBook *book)
gboolean e_book_check_revision(EBook *book , const gchar* revision)

and

gchar* e_cal_get_revision(ECal *ecal)
gboolean e_cal_check_revision(ECal ecal, const gchar revision)

The semantic would be this:
* If the revision string is empty or different, then the content of the database '''might''' have changed. A more detailed check is necessary to determine if it has really changed. This will be the case for non-local storage.
* If it is non-empty and the same as before, the nothing has changed. No further check will be necessary.

== contacts: store PHOTO data as plain files ==

Photos should be transferred over D-Bus via a filepath to a local file rather than transferring the actual photo data over D-Bus. The VCard format already allows this option (See the PHOTO property). We must be careful to keep these local files in sync and to remove them when contacts are removed, and to restrict access to the files from other processes.

There should be utility code available to libebook users to convert between the different representations.

Goals:
# Apps should be able to create photo files without ever passing the data to libebook.
# External PHOTO file and corresponding contact must be kept in sync:
** If a contact is deleted, the photo also needs to be removed.
** If a photo file is created and then storing the contact fails, who removes the file?
# Access to photo data must be limited to processes which have the right permissions.

Proposal:
* For each ~/.local/share/evolution/foo calendar, store photos in <photo path>=~/.local/share/evolution/foo/photos, with same permissions.
* New photo files are created with a [http://en.wikipedia.org/wiki/Universally_unique_identifier UUID] as base name and suffix according to the file content.
* If a contact is deleted and has a PHOTO;VALUE=uri:file://<photo path>/*, delete that file.
* For apps which want to create the photo file directly:
** Apps get a file descriptor and the path from a new libebook API.
** They own the file until the EDS server acknowledges the successful storing of a contact with a matching PHOTO;VALUE=uri:file property. If that fails, the app must remove the file.
* Add new API calls to libebook which split out resp. include data inline.
** Must track ownership of file after splitting it out and before storing - might never get stored!

Open:
* Should there be automatic garbage collection of stale photos in case something went wrong and a photo wasn't properly deleted?
* Trust TYPE=JPEG/PNG/... or look into file content to determine type?

Architecture/planning/evolution-data-server/eds-improvements

2011-06-08T10:03:26Z

Murrayc: More efficient access to meta data for change tracking + atomic updates: Split and clarify slightly.

This page describes improvements that are desirable or even essential for good performance of Evolution Data Server (EDS) as PIM storage in MeeGo.

All of these changes should be developed for the EDS master branch, reviewed by upstream maintainers and then back-ported to EDS 2.32, the version in MeeGo.

== content protection ==

Only authorized applications (= processes) with the necessary permissions should be able to read/write data. Exact permissions and mechanisms for verifying them to be defined as part of the (revised?) MeeGo architecture.

This can be added as follows:
# lock down file permissions
# add permission checks to EDS D-Bus servers

== change tracking ==

EDS is based on the model/view/controller principle: an application opens a database (calendar or addressbook) and requests a view with server-side filtering, then the application is sent:
# all data covered by the view
# the "all data sent" signal
# changes (add/update/remove) while the view exists

Applications modify data with asynchronous requests (controller) and keep showing the old data until the change notification comes in. The exact changes made to the data are determined by the server (for example when meeting invitations are sent).

QtContacts and KCalCore do not have such a strict model. With QtContacts the app simply loads and stores contacts and is responsible for maintaining its own view. Changes are sent for changes made after the database was opened. KCalCore is based on the concept of loading a calendar (partially) into the client, modifying it there, then writing it back. Changes are also sent while the storage is open.

TODO:

=== Suppress unnecessary data transfer ===

EDS currently sends all data when a view is requested, which is unnecessary when the client is only interested in change notifications. An extension to the EDS query language could optionally suppresses that sending of existing data when the view is opened, to be implemented only for the local contacts and calendar backends. This would allow easier integration with the QContact API.

The Fremantle EDS fork also had a “freezable” (static) capability for libebook, which should probably be investigated. See, for instance: http://maemo.org/api_refs/5.0/5.0-final/libebook/EBookView.html#e-book-view-set-freezable

One possibility for adding the flag is to (ab)use the existing query language with a match for special fields:
* in the client: query = e_book_query_orv(e_book_query_field_exists("X-EDS-QUERY-FLAG-SKIP-DATA-DUMP"), <real query>, NULL);
* in the server: "if (query is OR-Query and first clause is exists("X-EDS-QUERY-FLAG-SKIP-DATA-DUMP")) { skip_data_dump = TRUE; query = second clause; }

=== Return UIDs only ===

The e_book_get_book_view() function takes a list of requested fields, to avoid returning unnecessary data, but this is apparently ignored for the local contacts backend. It would anyway require extra parsing and generation of smaller VCards, losing some possible advantages from delayed parsing of VCards.

However, the UID is already stored separately for use as the primary key of the BDB database, so it would be worthwhile to support that particular requested field. This would allow easier integration with the QContact API. It would also offer much better performance in cases where the client already has the data but just wants to discover what subset of that data meets a certain criteria.

== optional parsing of ebook data in client ==

libebook provides different functionality:
# exchange data as vCard with EDS daemon
# parse items and wrap them in a C API

QtContacts replaces the second part. An obvious performance improvement is to only do the second part if the additional C APIs are really called. The following calls then transfer data without any additional parsing in libebook:
* e_book_get_contact()/asynchronous read/view + e_vcard_to_string()
* e_contact_new_from_vcard() + e_book_add_contact()/e_book_commit_contact()/asynchronous storing

Note that libecal (for iCalendar) can already do this, though libebook can't yet.

TODO:
# add delayed parsing to libebook, triggered by API call which needs parsed data

PROBLEM:
# If delayed parsing fails, how can error be reported?
# Should delayed parsing be done always (e_vcard_to_string())) or only if requested? How would that be configured by a libebook user?

== More efficient access to meta data for change tracking + atomic updates ==

This was discussed a while ago: [[http://www.mail-archive.com/evolution-hackers@gnome.org/msg03029.html
[Evolution-hackers] concurrent modifications of items in GUI and EDS database]]

Data synchronization only needs to know which items exist (list of local IDs) and which revision of each item is currently stored. Revisions can be identified with a string which changes for each modification. Restoring an older revision should (but doesn't have to) restore the revision string.

For calendar, UID+RECURRENCE-ID and LAST-MODIFIED can be used. For contacts, UID + REV.

By comparing a stored list against the current one, changes can be detected without relying on journals in the daemon or comparison of time stamps. Journals are hard to maintain (how many of them, how long?). Comparison of time stamps has problems when changes are allowed while a sync runs and assumes that system time is linear (which is not always true).

As an additional use case, local ID + revision string could be used to prevent overwriting stale data in the server, as in HTTP PUT eTag checks.

TODO:

The Maemo Fremantle port of EDS, which is of course already open source, apparently has some solution for this, according to Mathias Hasselman, so that should be investigated first.

However, this should probably be made possible by:

# Extending the EDS query language with a flag that limits the delivered data to just:
## UID and REV for libebook (REV is in VCard 3.0)
## UID+RECURRENCE-ID+LAST-MODIFIED, for libecal
# Implementing these queries for the local contact and calendar backends. The query would be rejected, with an error, for other backends.

== Atomic Updates ==

Later, implement atomic changes as discussed in the mail thread: [[http://www.mail-archive.com/evolution-hackers@gnome.org/msg03029.html

== quick check for "data changed" ==

Even quicker than listing items and comparing their revisions strings would be a comparison of a revision string for the whole database. The semantic would be this:
* If the revision string is empty or different, then the content of the database '''might''' have changed. A more detailed check is necessary to determine that.
* If it is non-empty and the same as before, nothing has changed. No further check necessary.

The modification time stamp of the .ics or .db files could be used. Probably need a new "query property" API. e_cal_get_static_capability() only returns bool. Same with e_book_get_static_capabilities()/e_book_check_static_capability().

== contacts: store PHOTO data as plain files ==

Photos should be transferred over D-Bus via a filepath to a local file rather than transferring the actual photo data over D-Bus. The VCard format already allows this option (See the PHOTO property). We must be careful to keep these local files in sync and to remove them when contacts are removed, and to restrict access to the files from other processes.

There should be utility code available to libebook users to convert between the different representations.

Goals:
# Apps should be able to create photo files without ever passing the data to libebook.
# External PHOTO file and corresponding contact must be kept in sync:
** If a contact is deleted, the photo also needs to be removed.
** If a photo file is created and then storing the contact fails, who removes the file?
# Access to photo data must be limited to processes which have the right permissions.

Proposal:
* For each ~/.local/share/evolution/foo calendar, store photos in <photo path>=~/.local/share/evolution/foo/photos, with same permissions.
* New photo files are created with a [http://en.wikipedia.org/wiki/Universally_unique_identifier UUID] as base name and suffix according to the file content.
* If a contact is deleted and has a PHOTO;VALUE=uri:file://<photo path>/*, delete that file.
* For apps which want to create the photo file directly:
** Apps get a file descriptor and the path from a new libebook API.
** They own the file until the EDS server acknowledges the successful storing of a contact with a matching PHOTO;VALUE=uri:file property. If that fails, the app must remove the file.
* Add new API calls to libebook which split out resp. include data inline.
** Must track ownership of file after splitting it out and before storing - might never get stored!

Open:
* Should there be automatic garbage collection of stale photos in case something went wrong and a photo wasn't properly deleted?
* Trust TYPE=JPEG/PNG/... or look into file content to determine type?

Architecture/planning/evolution-data-server/eds-improvements

2011-06-08T09:54:29Z

Murrayc: client parsing in libical is not actually necessary. While mplementing KCal-EDS, Patrick Ohly found that the parsing of iCalendar 2.0 data into libical can be done in libecal without causing overhead

This page describes improvements that are desirable or even essential for good performance of Evolution Data Server (EDS) as PIM storage in MeeGo.

All of these changes should be developed for the EDS master branch, reviewed by upstream maintainers and then back-ported to EDS 2.32, the version in MeeGo.

== content protection ==

Only authorized applications (= processes) with the necessary permissions should be able to read/write data. Exact permissions and mechanisms for verifying them to be defined as part of the (revised?) MeeGo architecture.

This can be added as follows:
# lock down file permissions
# add permission checks to EDS D-Bus servers

== change tracking ==

EDS is based on the model/view/controller principle: an application opens a database (calendar or addressbook) and requests a view with server-side filtering, then the application is sent:
# all data covered by the view
# the "all data sent" signal
# changes (add/update/remove) while the view exists

Applications modify data with asynchronous requests (controller) and keep showing the old data until the change notification comes in. The exact changes made to the data are determined by the server (for example when meeting invitations are sent).

QtContacts and KCalCore do not have such a strict model. With QtContacts the app simply loads and stores contacts and is responsible for maintaining its own view. Changes are sent for changes made after the database was opened. KCalCore is based on the concept of loading a calendar (partially) into the client, modifying it there, then writing it back. Changes are also sent while the storage is open.

TODO:

=== Suppress unnecessary data transfer ===

EDS currently sends all data when a view is requested, which is unnecessary when the client is only interested in change notifications. An extension to the EDS query language could optionally suppresses that sending of existing data when the view is opened, to be implemented only for the local contacts and calendar backends. This would allow easier integration with the QContact API.

The Fremantle EDS fork also had a “freezable” (static) capability for libebook, which should probably be investigated. See, for instance: http://maemo.org/api_refs/5.0/5.0-final/libebook/EBookView.html#e-book-view-set-freezable

One possibility for adding the flag is to (ab)use the existing query language with a match for special fields:
* in the client: query = e_book_query_orv(e_book_query_field_exists("X-EDS-QUERY-FLAG-SKIP-DATA-DUMP"), <real query>, NULL);
* in the server: "if (query is OR-Query and first clause is exists("X-EDS-QUERY-FLAG-SKIP-DATA-DUMP")) { skip_data_dump = TRUE; query = second clause; }

=== Return UIDs only ===

The e_book_get_book_view() function takes a list of requested fields, to avoid returning unnecessary data, but this is apparently ignored for the local contacts backend. It would anyway require extra parsing and generation of smaller VCards, losing some possible advantages from delayed parsing of VCards.

However, the UID is already stored separately for use as the primary key of the BDB database, so it would be worthwhile to support that particular requested field. This would allow easier integration with the QContact API. It would also offer much better performance in cases where the client already has the data but just wants to discover what subset of that data meets a certain criteria.

== optional parsing of ebook data in client ==

libebook provides different functionality:
# exchange data as vCard with EDS daemon
# parse items and wrap them in a C API

QtContacts replaces the second part. An obvious performance improvement is to only do the second part if the additional C APIs are really called. The following calls then transfer data without any additional parsing in libebook:
* e_book_get_contact()/asynchronous read/view + e_vcard_to_string()
* e_contact_new_from_vcard() + e_book_add_contact()/e_book_commit_contact()/asynchronous storing

Note that libecal (for iCalendar) can already do this, though libebook can't yet.

TODO:
# add delayed parsing to libebook, triggered by API call which needs parsed data

PROBLEM:
# If delayed parsing fails, how can error be reported?
# Should delayed parsing be done always (e_vcard_to_string())) or only if requested? How would that be configured by a libebook user?

== more efficient access to meta data for change tracking + atomic updates ==

This was discussed a while ago: [[http://www.mail-archive.com/evolution-hackers@gnome.org/msg03029.html
[Evolution-hackers] concurrent modifications of items in GUI and EDS database]]

Data synchronization only needs to know which items exist (list of local IDs) and which revision of each item is currently stored. Revisions can be identified with a string which changes for each modification. Restoring an older revision should (but doesn't have to) restore the revision string.

For calendar, UID+RECURRENCE-ID and LAST-MODIFIED can be used. For contacts, UID + REV.

By comparing a stored list against the current one, changes can be detected without relying on journals in the daemon or comparison of time stamps. Journals are hard to maintain (how many of them, how long?). Comparison of time stamps has problems when changes are allowed while a sync runs and assumes that system time is linear (which is not always true).

As an additional use case, local ID + revision string could be used to prevent overwriting stale data in the server, as in HTTP PUT eTag checks.

TODO:
# extend EDS query language with a flag that limits delivered data to just UID+RECURRENCE-ID+LAST-MODIFIED resp. UID+REV
# implement this query for local contact and calendar storage, reject with error in others
# later: implement atomic changes as discussed in the mail thread

== quick check for "data changed" ==

Even quicker than listing items and comparing their revisions strings would be a comparison of a revision string for the whole database. The semantic would be this:
* If the revision string is empty or different, then the content of the database '''might''' have changed. A more detailed check is necessary to determine that.
* If it is non-empty and the same as before, nothing has changed. No further check necessary.

The modification time stamp of the .ics or .db files could be used. Probably need a new "query property" API. e_cal_get_static_capability() only returns bool. Same with e_book_get_static_capabilities()/e_book_check_static_capability().

== contacts: store PHOTO data as plain files ==

Photos should be transferred over D-Bus via a filepath to a local file rather than transferring the actual photo data over D-Bus. The VCard format already allows this option (See the PHOTO property). We must be careful to keep these local files in sync and to remove them when contacts are removed, and to restrict access to the files from other processes.

There should be utility code available to libebook users to convert between the different representations.

Goals:
# Apps should be able to create photo files without ever passing the data to libebook.
# External PHOTO file and corresponding contact must be kept in sync:
** If a contact is deleted, the photo also needs to be removed.
** If a photo file is created and then storing the contact fails, who removes the file?
# Access to photo data must be limited to processes which have the right permissions.

Proposal:
* For each ~/.local/share/evolution/foo calendar, store photos in <photo path>=~/.local/share/evolution/foo/photos, with same permissions.
* New photo files are created with a [http://en.wikipedia.org/wiki/Universally_unique_identifier UUID] as base name and suffix according to the file content.
* If a contact is deleted and has a PHOTO;VALUE=uri:file://<photo path>/*, delete that file.
* For apps which want to create the photo file directly:
** Apps get a file descriptor and the path from a new libebook API.
** They own the file until the EDS server acknowledges the successful storing of a contact with a matching PHOTO;VALUE=uri:file property. If that fails, the app must remove the file.
* Add new API calls to libebook which split out resp. include data inline.
** Must track ownership of file after splitting it out and before storing - might never get stored!

Open:
* Should there be automatic garbage collection of stale photos in case something went wrong and a photo wasn't properly deleted?
* Trust TYPE=JPEG/PNG/... or look into file content to determine type?

Architecture/planning/evolution-data-server/eds-improvements

2011-06-08T09:48:43Z

Murrayc: Contacts: store PHOTO data as plain files: Clarify the text slightly.

This page describes improvements that are desirable or even essential for good performance of Evolution Data Server (EDS) as PIM storage in MeeGo.

All of these changes should be developed for the EDS master branch, reviewed by upstream maintainers and then back-ported to EDS 2.32, the version in MeeGo.

== content protection ==

Only authorized applications (= processes) with the necessary permissions should be able to read/write data. Exact permissions and mechanisms for verifying them to be defined as part of the (revised?) MeeGo architecture.

This can be added as follows:
# lock down file permissions
# add permission checks to EDS D-Bus servers

== change tracking ==

EDS is based on the model/view/controller principle: an application opens a database (calendar or addressbook) and requests a view with server-side filtering, then the application is sent:
# all data covered by the view
# the "all data sent" signal
# changes (add/update/remove) while the view exists

Applications modify data with asynchronous requests (controller) and keep showing the old data until the change notification comes in. The exact changes made to the data are determined by the server (for example when meeting invitations are sent).

QtContacts and KCalCore do not have such a strict model. With QtContacts the app simply loads and stores contacts and is responsible for maintaining its own view. Changes are sent for changes made after the database was opened. KCalCore is based on the concept of loading a calendar (partially) into the client, modifying it there, then writing it back. Changes are also sent while the storage is open.

TODO:

=== Suppress unnecessary data transfer ===

EDS currently sends all data when a view is requested, which is unnecessary when the client is only interested in change notifications. An extension to the EDS query language could optionally suppresses that sending of existing data when the view is opened, to be implemented only for the local contacts and calendar backends. This would allow easier integration with the QContact API.

The Fremantle EDS fork also had a “freezable” (static) capability for libebook, which should probably be investigated. See, for instance: http://maemo.org/api_refs/5.0/5.0-final/libebook/EBookView.html#e-book-view-set-freezable

One possibility for adding the flag is to (ab)use the existing query language with a match for special fields:
* in the client: query = e_book_query_orv(e_book_query_field_exists("X-EDS-QUERY-FLAG-SKIP-DATA-DUMP"), <real query>, NULL);
* in the server: "if (query is OR-Query and first clause is exists("X-EDS-QUERY-FLAG-SKIP-DATA-DUMP")) { skip_data_dump = TRUE; query = second clause; }

=== Return UIDs only ===

The e_book_get_book_view() function takes a list of requested fields, to avoid returning unnecessary data, but this is apparently ignored for the local contacts backend. It would anyway require extra parsing and generation of smaller VCards, losing some possible advantages from delayed parsing of VCards.

However, the UID is already stored separately for use as the primary key of the BDB database, so it would be worthwhile to support that particular requested field. This would allow easier integration with the QContact API. It would also offer much better performance in cases where the client already has the data but just wants to discover what subset of that data meets a certain criteria.

== optional parsing of data in client ==

libebook/ecal provide different functionality:
# exchange data as vCard/iCalendar with EDS daemon
# parse items and wrap them in a C API

QtContacts and KCalCore replace the second part. An obvious performance improvement is to only do the second part if the additional C APIs are really called. The following calls then transfer data without any additional parsing in libebook/libecal:
* e_book_get_contact()/asynchronous read/view + e_vcard_to_string()
* e_contact_new_from_vcard() + e_book_add_contact()/e_book_commit_contact()/asynchronous storing
* e_cal_get_object_list_as_comp()/asynchronous read/view + e_cal_get_component_as_string()
* icalcomponent_new_from_string() + e_cal_modify_object[_with_mod]()/e_cal_create_object()
* e_cal_get_object_list_as_comp()/e_cal_get_object()/asynchronous view + e_cal_get_component_as_string()

TODO:
# add delayed parsing to libebook, triggered by API call which needs parsed data
# add delayed parsing to libical, again triggered by API call

PROBLEM:
# If delayed parsing fails, how can error be reported?
# Should delayed parsing be done always (e_vcard_to_string()/icalcomponent_new_from_string()) or only if requested? How would that be configured by a libebook/libecal user?

== more efficient access to meta data for change tracking + atomic updates ==

This was discussed a while ago: [[http://www.mail-archive.com/evolution-hackers@gnome.org/msg03029.html
[Evolution-hackers] concurrent modifications of items in GUI and EDS database]]

Data synchronization only needs to know which items exist (list of local IDs) and which revision of each item is currently stored. Revisions can be identified with a string which changes for each modification. Restoring an older revision should (but doesn't have to) restore the revision string.

For calendar, UID+RECURRENCE-ID and LAST-MODIFIED can be used. For contacts, UID + REV.

By comparing a stored list against the current one, changes can be detected without relying on journals in the daemon or comparison of time stamps. Journals are hard to maintain (how many of them, how long?). Comparison of time stamps has problems when changes are allowed while a sync runs and assumes that system time is linear (which is not always true).

As an additional use case, local ID + revision string could be used to prevent overwriting stale data in the server, as in HTTP PUT eTag checks.

TODO:
# extend EDS query language with a flag that limits delivered data to just UID+RECURRENCE-ID+LAST-MODIFIED resp. UID+REV
# implement this query for local contact and calendar storage, reject with error in others
# later: implement atomic changes as discussed in the mail thread

== quick check for "data changed" ==

Even quicker than listing items and comparing their revisions strings would be a comparison of a revision string for the whole database. The semantic would be this:
* If the revision string is empty or different, then the content of the database '''might''' have changed. A more detailed check is necessary to determine that.
* If it is non-empty and the same as before, nothing has changed. No further check necessary.

The modification time stamp of the .ics or .db files could be used. Probably need a new "query property" API. e_cal_get_static_capability() only returns bool. Same with e_book_get_static_capabilities()/e_book_check_static_capability().

== contacts: store PHOTO data as plain files ==

Photos should be transferred over D-Bus via a filepath to a local file rather than transferring the actual photo data over D-Bus. The VCard format already allows this option (See the PHOTO property). We must be careful to keep these local files in sync and to remove them when contacts are removed, and to restrict access to the files from other processes.

There should be utility code available to libebook users to convert between the different representations.

Goals:
# Apps should be able to create photo files without ever passing the data to libebook.
# External PHOTO file and corresponding contact must be kept in sync:
** If a contact is deleted, the photo also needs to be removed.
** If a photo file is created and then storing the contact fails, who removes the file?
# Access to photo data must be limited to processes which have the right permissions.

Proposal:
* For each ~/.local/share/evolution/foo calendar, store photos in <photo path>=~/.local/share/evolution/foo/photos, with same permissions.
* New photo files are created with a [http://en.wikipedia.org/wiki/Universally_unique_identifier UUID] as base name and suffix according to the file content.
* If a contact is deleted and has a PHOTO;VALUE=uri:file://<photo path>/*, delete that file.
* For apps which want to create the photo file directly:
** Apps get a file descriptor and the path from a new libebook API.
** They own the file until the EDS server acknowledges the successful storing of a contact with a matching PHOTO;VALUE=uri:file property. If that fails, the app must remove the file.
* Add new API calls to libebook which split out resp. include data inline.
** Must track ownership of file after splitting it out and before storing - might never get stored!

Open:
* Should there be automatic garbage collection of stale photos in case something went wrong and a photo wasn't properly deleted?
* Trust TYPE=JPEG/PNG/... or look into file content to determine type?

Architecture/planning/evolution-data-server/eds-improvements

2011-06-08T09:45:08Z

Murrayc: change tracking: Make the work to be done clearer.

This page describes improvements that are desirable or even essential for good performance of Evolution Data Server (EDS) as PIM storage in MeeGo.

All of these changes should be developed for the EDS master branch, reviewed by upstream maintainers and then back-ported to EDS 2.32, the version in MeeGo.

== content protection ==

Only authorized applications (= processes) with the necessary permissions should be able to read/write data. Exact permissions and mechanisms for verifying them to be defined as part of the (revised?) MeeGo architecture.

This can be added as follows:
# lock down file permissions
# add permission checks to EDS D-Bus servers

== change tracking ==

EDS is based on the model/view/controller principle: an application opens a database (calendar or addressbook) and requests a view with server-side filtering, then the application is sent:
# all data covered by the view
# the "all data sent" signal
# changes (add/update/remove) while the view exists

Applications modify data with asynchronous requests (controller) and keep showing the old data until the change notification comes in. The exact changes made to the data are determined by the server (for example when meeting invitations are sent).

QtContacts and KCalCore do not have such a strict model. With QtContacts the app simply loads and stores contacts and is responsible for maintaining its own view. Changes are sent for changes made after the database was opened. KCalCore is based on the concept of loading a calendar (partially) into the client, modifying it there, then writing it back. Changes are also sent while the storage is open.

TODO:

=== Suppress unnecessary data transfer ===

EDS currently sends all data when a view is requested, which is unnecessary when the client is only interested in change notifications. An extension to the EDS query language could optionally suppresses that sending of existing data when the view is opened, to be implemented only for the local contacts and calendar backends. This would allow easier integration with the QContact API.

The Fremantle EDS fork also had a “freezable” (static) capability for libebook, which should probably be investigated. See, for instance: http://maemo.org/api_refs/5.0/5.0-final/libebook/EBookView.html#e-book-view-set-freezable

One possibility for adding the flag is to (ab)use the existing query language with a match for special fields:
* in the client: query = e_book_query_orv(e_book_query_field_exists("X-EDS-QUERY-FLAG-SKIP-DATA-DUMP"), <real query>, NULL);
* in the server: "if (query is OR-Query and first clause is exists("X-EDS-QUERY-FLAG-SKIP-DATA-DUMP")) { skip_data_dump = TRUE; query = second clause; }

=== Return UIDs only ===

The e_book_get_book_view() function takes a list of requested fields, to avoid returning unnecessary data, but this is apparently ignored for the local contacts backend. It would anyway require extra parsing and generation of smaller VCards, losing some possible advantages from delayed parsing of VCards.

However, the UID is already stored separately for use as the primary key of the BDB database, so it would be worthwhile to support that particular requested field. This would allow easier integration with the QContact API. It would also offer much better performance in cases where the client already has the data but just wants to discover what subset of that data meets a certain criteria.

== optional parsing of data in client ==

libebook/ecal provide different functionality:
# exchange data as vCard/iCalendar with EDS daemon
# parse items and wrap them in a C API

QtContacts and KCalCore replace the second part. An obvious performance improvement is to only do the second part if the additional C APIs are really called. The following calls then transfer data without any additional parsing in libebook/libecal:
* e_book_get_contact()/asynchronous read/view + e_vcard_to_string()
* e_contact_new_from_vcard() + e_book_add_contact()/e_book_commit_contact()/asynchronous storing
* e_cal_get_object_list_as_comp()/asynchronous read/view + e_cal_get_component_as_string()
* icalcomponent_new_from_string() + e_cal_modify_object[_with_mod]()/e_cal_create_object()
* e_cal_get_object_list_as_comp()/e_cal_get_object()/asynchronous view + e_cal_get_component_as_string()

TODO:
# add delayed parsing to libebook, triggered by API call which needs parsed data
# add delayed parsing to libical, again triggered by API call

PROBLEM:
# If delayed parsing fails, how can error be reported?
# Should delayed parsing be done always (e_vcard_to_string()/icalcomponent_new_from_string()) or only if requested? How would that be configured by a libebook/libecal user?

== more efficient access to meta data for change tracking + atomic updates ==

This was discussed a while ago: [[http://www.mail-archive.com/evolution-hackers@gnome.org/msg03029.html
[Evolution-hackers] concurrent modifications of items in GUI and EDS database]]

Data synchronization only needs to know which items exist (list of local IDs) and which revision of each item is currently stored. Revisions can be identified with a string which changes for each modification. Restoring an older revision should (but doesn't have to) restore the revision string.

For calendar, UID+RECURRENCE-ID and LAST-MODIFIED can be used. For contacts, UID + REV.

By comparing a stored list against the current one, changes can be detected without relying on journals in the daemon or comparison of time stamps. Journals are hard to maintain (how many of them, how long?). Comparison of time stamps has problems when changes are allowed while a sync runs and assumes that system time is linear (which is not always true).

As an additional use case, local ID + revision string could be used to prevent overwriting stale data in the server, as in HTTP PUT eTag checks.

TODO:
# extend EDS query language with a flag that limits delivered data to just UID+RECURRENCE-ID+LAST-MODIFIED resp. UID+REV
# implement this query for local contact and calendar storage, reject with error in others
# later: implement atomic changes as discussed in the mail thread

== quick check for "data changed" ==

Even quicker than listing items and comparing their revisions strings would be a comparison of a revision string for the whole database. The semantic would be this:
* If the revision string is empty or different, then the content of the database '''might''' have changed. A more detailed check is necessary to determine that.
* If it is non-empty and the same as before, nothing has changed. No further check necessary.

The modification time stamp of the .ics or .db files could be used. Probably need a new "query property" API. e_cal_get_static_capability() only returns bool. Same with e_book_get_static_capabilities()/e_book_check_static_capability().

== contacts: store PHOTO data as plain files ==

Transfering PHOTO data over D-Bus as base64 encoded data in each vCard has obvious performance issues. Instead the PHOTO property should be a link to a local file (supported by vCard), with utility code available to libebook users to convert between the different representations.

Goals:
# Apps should be able to create photo files without ever passing the data to libebook.
# External PHOTO file and corresponding contact must be kept in sync:
** If a contact is deleted, the photo also needs to be removed.
** If a photo file is created and then storing the contact fails, who removes the file?
# Access to photo data must be limited to processes which have the right permissions.

Proposal:
* For each ~/.local/share/evolution/foo calendar, store photos in <photo path>=~/.local/share/evolution/foo/photos, with same permissions.
* New photo files are created with a [http://en.wikipedia.org/wiki/Universally_unique_identifier UUID] as base name and suffix according to the file content.
* If a contact is deleted and has a PHOTO;VALUE=uri:file://<photo path>/*, delete that file.
* For apps which want to create the photo file directly:
** Apps get a file descriptor and the path from a new libebook API.
** They own the file until the EDS server acknowledges the successful storing of a contact with a matching PHOTO;VALUE=uri:file property. If that fails, the app must remove the file.
* Add new API calls to libebook which split out resp. include data inline.
** Must track ownership of file after splitting it out and before storing - might never get stored!

Open:
* Should there be automatic garbage collection of stale photos in case something went wrong and a photo wasn't properly deleted?
* Trust TYPE=JPEG/PNG/... or look into file content to determine type?