Meego Wiki
Views

Architecture/planning/evolution-data-server/email-design

From MeeGo wiki
< Architecture | planning/evolution-data-server(Difference between revisions)
Jump to: navigation, search
(Created page with "EMailFactory will be a dbus daemon, that runs camel session. The camel session controls the providers of IMAP/POP/Exchange/* and keeps checking emails, and serves CamelStore/Came…")
(→QtMobility support)
 
(19 intermediate revisions not shown)
Line 1: Line 1:
-
EMailFactory will be a dbus daemon, that runs camel session. The camel session controls the providers of IMAP/POP/Exchange/* and keeps checking emails, and serves CamelStore/CamelFolder/CamelMimeMessage/* over DBUS to its clients. Below is a high level architecture diagram of how the various pieces fit together.
+
EMailFactory will be a dbus daemon, that runs camel session. The camel session controls the providers of IMAP/POP/Exchange/* and keeps checking emails, and serves CamelStore/CamelFolder/CamelMimeMessage/* over DBUS to its clients. Below is a high level architecture diagram of how the various pieces fit together, displayed below.
-
[[File:Email-daemon-design-1.png]]
+
[[File:Email-daemon-design-complete-1.png]]
 +
 
 +
 
 +
 
 +
== Source  ==
 +
 
 +
The Camel Daemon code is now in gnome git, in the email-factory branch.<br>
 +
http://git.gnome.org/browse/evolution-data-server/log/?h=email-factory has the code the for the daemon:<br>
 +
<br>
 +
The DBus API currently provided by the Daemon is below. In a high level, it is a async wrapper around the camel's session/folder/store API.<br>
 +
http://git.gnome.org/browse/evolution-data-server/tree/mail/daemon/e-mail-data-folder.xml?h=email-factory<br>
 +
http://git.gnome.org/browse/evolution-data-server/tree/mail/daemon/e-mail-data-store.xml?h=email-factory<br>
 +
http://git.gnome.org/browse/evolution-data-server/tree/mail/daemon/e-mail-data-session.xml?h=email-factory<br>
 +
 
 +
== About Camel Daemon ==
 +
 
 +
Camel Daemon (The EMail factory) is a process that runs camel session. A high level picture of the daemon is below:
 +
[[File:Cameldesign.png]]
 +
 
 +
== MeeGo Tablet mail integration ==
 +
 
 +
* Integrate Account setup with EAccountList/EAccount in Gconf.
 +
It could be a challenge to verify accounts. As Evolution doesn't do that yet. It runs accounts and reports error if its not right.
 +
* Deeply integrate the camel daemon's dbus api into the email app
 +
Have the working C++ dbus interface for the Camel daemon, created working instances & marshaling code etc. Moved away the base objects of folderlistmodel, emailaccountlistmodel and emailmessagelist model from QMF baseclass to QAbstract*Model.  Implemented these models to pick up data from Camel daemon. Modified a bit of QML set the folder/account correctly.<br>
 +
Pending:
 +
 
 +
# Modify EmailAgent, in a way it send/receives mails to Camel
 +
# Modify EMailFeedModel to handle EmailMessage creation from Camel.
 +
# Integrate CamelMimeMessage & mime part hierarchy for things like attachment handling, plain text and html body.
 +
# Composer integration & sending mails.
 +
# Mail operations handling: like create/delete mail/folder, reply/forward, mark read/unread, sort, search.
 +
<br>
 +
* Camel API:
 +
Almost complete to suffice our need, have a test code that runs through all the APIs and work well so far. Expect a few minor changes here & there on demand.
 +
 
 +
Pending:
 +
# Write libcamelremote a wrapper around the DBus interface to provide sync/async apis for the camel Daemon.
 +
<br>
 +
* Camel Daemon:
 +
Works mostly, and is pretty stable than expected. But should spend some time & check on the CPU usage,dbus connection handling, and object clean ups.
 +
# DBus conneciton handling & object clean ups.
 +
# EMail preview support. Anjal used to extend evolution to add email preview of unquoted text. I should add <br>that  bit to Camel so that it can be served to the tablet email app.
 +
# Gtk free the daemon. Specially the password dialog. Add signal to integrate with clients for password handling. Add better error handling and progress reporting.
 +
<br>
 +
 
 +
== QtMobility support ==
 +
 
 +
3rd party apps will use QtMobility to interface with email framework than the raw DBus apis.
 +
 
 +
The [http://doc.qt.nokia.com/qtmobility-1.2/qtmessaging.html QtMessaging] module is a QtMobility module providing access to messaging services.
 +
 
 +
In order to provide implementation of QtMessaging interfaces a new dynamic library should be created. <br>
 +
The library should be loaded into email client application process. <br>
 +
The library should expose QtMessaging interfaces and map QtMessaging api to Camel daemon dBus api.<br>
 +
Possibility of direct read from Camel local storage shall be discussed.
 +
 
 +
Short analysis of how QtMessaging interfaces can be implemented via [http://www.go-evolution.org/Camel Camel] back-end is shown on table below.
 +
{| border="1"
 +
! QtMessaging interface
 +
! Implementation in Camel back-end
 +
! Missing features
 +
! Solution
 +
|-
 +
|QMessageDataComparator
 +
|Does not exist
 +
|Should be implemented from scratch
 +
|Simple aux object. QMF implementation can be used.
 +
 
 +
|-
 +
|QMessage
 +
|Camel.MessageInfo <br> CamelMessageInfoBase
 +
|Message flags are different. Some properties are missing but can be calculated.
 +
|Camel.MessageInfo can be mapped to QMessage.
 +
 
 +
|-
 +
|QMessageAccount
 +
|EAccount
 +
|Needed functionality is supported.
 +
|EAccount can be mapped to QMessageAccount
 +
|-
 +
|QMessageAccountFilter
 +
|Does not exist
 +
|Should be implemented from scratch
 +
|QtMessaging allows not support it or support partially.<br> Should store search parameters and QMessageManager class should consider them while accounts querying. Implementation  of  logical operators is the most difficult part..
 +
 
 +
|-
 +
|QMessageAccountId
 +
|Eaccount->uid
 +
|Should be implemented from scratch
 +
|Easy-to-implement object. Will just contain 'Eaccount->uid' inside
 +
 
 +
|-
 +
|QMessageAccountSortOrder
 +
|Does not exist
 +
|Should be implemented from scratch
 +
|QtMessaging allows not support it or support partially.<br> Easy-to-implement object. Should store sorting mode and QMessageManager class must consider it while  account querying.
 +
 
 +
|-
 +
|QMessageAddress
 +
|CamelInternetAddress
 +
|Needed functionality (email address parsing) is supported.
 +
|CamelInternetAddress can be mapped to QMessageAddress (alternatively QMF code could be  used)
 +
 
 +
|-
 +
|QMessageContentContainer
 +
|Camel.MessageContentInfo
 +
|Needed functionality is supported.
 +
|Camel.MessageContentInfo can be mapped to QMessageContentContainer.
 +
 
 +
|-
 +
|QMessageContentContainerId
 +
|CamelMessageContentInfo->id
 +
|Should be implemented from scratch
 +
|Easy-to-implement object. Will contain 'CamelMessageContentInfo->id' inside
 +
 
 +
|-
 +
|QMessageFilter
 +
|Camel.FolderSearch
 +
|
 +
* Inter-folder search is missing. (Srini, could you clarify whether vFolder can solve it?).
 +
* Searching of messages by parent/ancestors folder id is missing.
 +
* Searching of messages by parent account id is missing
 +
|QtMessaging allows not support it or support partially.<br> QMessageFilter should obviously be mapped to ESExp (where it is possible).  Implementation  of logical operators should also fruit resulting  ESExp.<br> Challenging case here is combination of search via ESExp and self-implemented search (i.e. by ancestors folders).
 +
 
 +
|-
 +
|QMessageFolder
 +
|CamelFolder
 +
|Needed functionality is supported.
 +
|CamelFolder can be mapped to QMessageFolder.
 +
 
 +
|-
 +
|QMessageFolderFilter
 +
|Does not exist
 +
|Should be implemented from scratch
 +
|QtMessaging allows not support it or support partially.<br>
 +
Should store search parameters and QMessageManager class should consider them while folder querying. Implementation of logical operators is the most difficult part.
 +
 
 +
|-
 +
|QMessageFolderId
 +
|CamelFolder->full_name
 +
|Should be implemented from scratch
 +
|Easy-to-implement object. Will contain 'CamelFolder->full_name' inside.
 +
 
 +
|-
 +
|QMessageFolderSortOrder
 +
|Does not exist
 +
|Should be implemented from scratch
 +
|QtMessaging allows not support it or support partially.<br>
 +
Should store sort parameters and QMessageManager class should consider them while folder querying.
 +
 
 +
|-
 +
|QMessageId
 +
|CamelMessageInfo->uid
 +
|Should be implemented from scratch
 +
|Easy-to-implement object. Will contain 'CamelMessageInfo->uid' inside.
 +
 
 +
|-
 +
|QMessageManager
 +
|Camel.Store (folders and messages),<br> EAccountList (e-mail accounts)
 +
|QMessageManager has sophisticated methods which query/add/remove/update  accounts, folders and messages using filtering and sorting.<br> Most of this functionality does not exist in Camel back-end.
 +
|Camel.Search can be used for searching of messages within a folder or a set of folders.<br>
 +
Operations with folders and accounts are to be implemented probably by checking and sorting of all the existing items and this can be time-consuming.
 +
The implementation of unsupported storage operations can be located either inside the library itself or (and much more preferable) inside Camel daemon.
 +
 
 +
 
 +
 
 +
|-
 +
|QMessageService
 +
|Camel.IMAP<br>Camel.POP3<br>Camel.SMTP
 +
|Needed functionality is supported.
 +
|There should be a set of classes with common interface on top of Camel protocol providers.
 +
 
 +
 
 +
|-
 +
|QMessageSortOrder
 +
|Does not exist
 +
|Should be implemented from scratch
 +
|QtMessaging allows not support it or support partially.<br>
 +
Should store sort parameters and QMessageManager class should consider them while message querying.
 +
 
 +
|}
 +
 
 +
=== Notes ===
 +
 
 +
* QMessageManager  (the main QtMessaging interface ) is wholly based on top of QMF storage class implementation and its methods are supposed to be converted to QMF database requests.<br> Camel local storage is completely different and implementation of many methods (i.e. folders and accounts operations) will have to be done ”manually”  by checking and/or sorting of all the existing items. Hence performance is affected.
 +
* Implementation of QMessageManager interface is supposed to provide inter-account message/folder querying (all the accounts in QMF share common DB) and this seems to be very expensive in our case as there is one storage per account in Camel.
 +
* All the methods in QMessageManager interface are synchronous and aimed to access local storage directly. Camel local storage does not support direct multi-process access so all the storage operations have to be processed through Camel daemon ipc interface. <br> Of course ipc access is much slower and performing it synchronously  (due to QMessageManager requirements) means that all the local storage operations on e-mail client side must be performed in a separate thread so that e-mail client UI is kept from freezing.  Consequences are the following:
 +
** QtMessaging implementation needs multi-threaded support
 +
** email client application is multi-threaded and overcomplicated (third party developers will not like it)
 +
** QMF has no multi-threaded support and all the existing applications invoke QtMessaging interface from the main thread => our implementation is incompatible
 +
** main thread in email client will communicate with working thread asynchronously (same way it could do with Camel daemon async ipc interface directly without necessity of multi-threaded mess)
 +
 
 +
=== Summary ===
 +
 
 +
* QtMessaging interfaces can be implemented with Camel back-end (it is not impossible)
 +
* Implementation of the QMessageManager interface will be extremely slow and hard-to-use.
 +
* Implementation of the QMessageManager interface will be incompatible with the existing applications.
 +
 
 +
=== Proposal ===
 +
 
 +
Asynchronous interface for local storage access is needed. Extending of QtMessaging with asynchronous message manager (i.e. QAsyncMessageManager) would be a nice solution.

Latest revision as of 14:03, 16 June 2011

EMailFactory will be a dbus daemon, that runs camel session. The camel session controls the providers of IMAP/POP/Exchange/* and keeps checking emails, and serves CamelStore/CamelFolder/CamelMimeMessage/* over DBUS to its clients. Below is a high level architecture diagram of how the various pieces fit together, displayed below.

Email-daemon-design-complete-1.png


Contents

Source

The Camel Daemon code is now in gnome git, in the email-factory branch.
http://git.gnome.org/browse/evolution-data-server/log/?h=email-factory has the code the for the daemon:

The DBus API currently provided by the Daemon is below. In a high level, it is a async wrapper around the camel's session/folder/store API.
http://git.gnome.org/browse/evolution-data-server/tree/mail/daemon/e-mail-data-folder.xml?h=email-factory
http://git.gnome.org/browse/evolution-data-server/tree/mail/daemon/e-mail-data-store.xml?h=email-factory
http://git.gnome.org/browse/evolution-data-server/tree/mail/daemon/e-mail-data-session.xml?h=email-factory

About Camel Daemon

Camel Daemon (The EMail factory) is a process that runs camel session. A high level picture of the daemon is below: Cameldesign.png

MeeGo Tablet mail integration

  • Integrate Account setup with EAccountList/EAccount in Gconf.

It could be a challenge to verify accounts. As Evolution doesn't do that yet. It runs accounts and reports error if its not right.

  • Deeply integrate the camel daemon's dbus api into the email app

Have the working C++ dbus interface for the Camel daemon, created working instances & marshaling code etc. Moved away the base objects of folderlistmodel, emailaccountlistmodel and emailmessagelist model from QMF baseclass to QAbstract*Model. Implemented these models to pick up data from Camel daemon. Modified a bit of QML set the folder/account correctly.
Pending:

  1. Modify EmailAgent, in a way it send/receives mails to Camel
  2. Modify EMailFeedModel to handle EmailMessage creation from Camel.
  3. Integrate CamelMimeMessage & mime part hierarchy for things like attachment handling, plain text and html body.
  4. Composer integration & sending mails.
  5. Mail operations handling: like create/delete mail/folder, reply/forward, mark read/unread, sort, search.


  • Camel API:

Almost complete to suffice our need, have a test code that runs through all the APIs and work well so far. Expect a few minor changes here & there on demand.

Pending:

  1. Write libcamelremote a wrapper around the DBus interface to provide sync/async apis for the camel Daemon.


  • Camel Daemon:

Works mostly, and is pretty stable than expected. But should spend some time & check on the CPU usage,dbus connection handling, and object clean ups.

  1. DBus conneciton handling & object clean ups.
  2. EMail preview support. Anjal used to extend evolution to add email preview of unquoted text. I should add
    that bit to Camel so that it can be served to the tablet email app.
  3. Gtk free the daemon. Specially the password dialog. Add signal to integrate with clients for password handling. Add better error handling and progress reporting.


QtMobility support

3rd party apps will use QtMobility to interface with email framework than the raw DBus apis.

The QtMessaging module is a QtMobility module providing access to messaging services.

In order to provide implementation of QtMessaging interfaces a new dynamic library should be created.
The library should be loaded into email client application process.
The library should expose QtMessaging interfaces and map QtMessaging api to Camel daemon dBus api.
Possibility of direct read from Camel local storage shall be discussed.

Short analysis of how QtMessaging interfaces can be implemented via Camel back-end is shown on table below.

QtMessaging interface Implementation in Camel back-end Missing features Solution
QMessageDataComparator Does not exist Should be implemented from scratch Simple aux object. QMF implementation can be used.
QMessage Camel.MessageInfo
CamelMessageInfoBase 		Message flags are different. Some properties are missing but can be calculated. Camel.MessageInfo can be mapped to QMessage.
QMessageAccount EAccount Needed functionality is supported. EAccount can be mapped to QMessageAccount
QMessageAccountFilter Does not exist Should be implemented from scratch QtMessaging allows not support it or support partially.
Should store search parameters and QMessageManager class should consider them while accounts querying. Implementation of logical operators is the most difficult part..
QMessageAccountId Eaccount->uid Should be implemented from scratch Easy-to-implement object. Will just contain 'Eaccount->uid' inside
QMessageAccountSortOrder Does not exist Should be implemented from scratch QtMessaging allows not support it or support partially.
Easy-to-implement object. Should store sorting mode and QMessageManager class must consider it while account querying.
QMessageAddress CamelInternetAddress Needed functionality (email address parsing) is supported. CamelInternetAddress can be mapped to QMessageAddress (alternatively QMF code could be used)
QMessageContentContainer Camel.MessageContentInfo Needed functionality is supported. Camel.MessageContentInfo can be mapped to QMessageContentContainer.
QMessageContentContainerId CamelMessageContentInfo->id Should be implemented from scratch Easy-to-implement object. Will contain 'CamelMessageContentInfo->id' inside
QMessageFilter Camel.FolderSearch
  • Inter-folder search is missing. (Srini, could you clarify whether vFolder can solve it?).
  • Searching of messages by parent/ancestors folder id is missing.
  • Searching of messages by parent account id is missing
QtMessaging allows not support it or support partially.
QMessageFilter should obviously be mapped to ESExp (where it is possible). Implementation of logical operators should also fruit resulting ESExp.
Challenging case here is combination of search via ESExp and self-implemented search (i.e. by ancestors folders).
QMessageFolder CamelFolder Needed functionality is supported. CamelFolder can be mapped to QMessageFolder.
QMessageFolderFilter Does not exist Should be implemented from scratch QtMessaging allows not support it or support partially.

Should store search parameters and QMessageManager class should consider them while folder querying. Implementation of logical operators is the most difficult part.

QMessageFolderId CamelFolder->full_name Should be implemented from scratch Easy-to-implement object. Will contain 'CamelFolder->full_name' inside.
QMessageFolderSortOrder Does not exist Should be implemented from scratch QtMessaging allows not support it or support partially.

Should store sort parameters and QMessageManager class should consider them while folder querying.

QMessageId CamelMessageInfo->uid Should be implemented from scratch Easy-to-implement object. Will contain 'CamelMessageInfo->uid' inside.
QMessageManager Camel.Store (folders and messages),
EAccountList (e-mail accounts) 		QMessageManager has sophisticated methods which query/add/remove/update accounts, folders and messages using filtering and sorting.
Most of this functionality does not exist in Camel back-end. 		Camel.Search can be used for searching of messages within a folder or a set of folders.

Operations with folders and accounts are to be implemented probably by checking and sorting of all the existing items and this can be time-consuming. The implementation of unsupported storage operations can be located either inside the library itself or (and much more preferable) inside Camel daemon.


QMessageService Camel.IMAP
Camel.POP3
Camel.SMTP 		Needed functionality is supported. There should be a set of classes with common interface on top of Camel protocol providers.


QMessageSortOrder Does not exist Should be implemented from scratch QtMessaging allows not support it or support partially.

Should store sort parameters and QMessageManager class should consider them while message querying.

Notes

  • QMessageManager (the main QtMessaging interface ) is wholly based on top of QMF storage class implementation and its methods are supposed to be converted to QMF database requests.
    Camel local storage is completely different and implementation of many methods (i.e. folders and accounts operations) will have to be done ”manually” by checking and/or sorting of all the existing items. Hence performance is affected.
  • Implementation of QMessageManager interface is supposed to provide inter-account message/folder querying (all the accounts in QMF share common DB) and this seems to be very expensive in our case as there is one storage per account in Camel.
  • All the methods in QMessageManager interface are synchronous and aimed to access local storage directly. Camel local storage does not support direct multi-process access so all the storage operations have to be processed through Camel daemon ipc interface.
    Of course ipc access is much slower and performing it synchronously (due to QMessageManager requirements) means that all the local storage operations on e-mail client side must be performed in a separate thread so that e-mail client UI is kept from freezing. Consequences are the following:
    • QtMessaging implementation needs multi-threaded support
    • email client application is multi-threaded and overcomplicated (third party developers will not like it)
    • QMF has no multi-threaded support and all the existing applications invoke QtMessaging interface from the main thread => our implementation is incompatible
    • main thread in email client will communicate with working thread asynchronously (same way it could do with Camel daemon async ipc interface directly without necessity of multi-threaded mess)

Summary

  • QtMessaging interfaces can be implemented with Camel back-end (it is not impossible)
  • Implementation of the QMessageManager interface will be extremely slow and hard-to-use.
  • Implementation of the QMessageManager interface will be incompatible with the existing applications.

Proposal

Asynchronous interface for local storage access is needed. Extending of QtMessaging with asynchronous message manager (i.e. QAsyncMessageManager) would be a nice solution.

Retrieved from "http://wiki.meego.com/Architecture/planning/evolution-data-server/email-design"
Personal tools