Evolution MAPI (Exchange 2007) Preview

Johnny sent out a mail on the Evolution list about the availability of the preview build of the Evolution MAPI provider. Read the complete thread, to know what works and what doesn’t and other known issues. Currently we have rpms for OpenSUSE 10.3 and Fedora only. We would be providing rpms for Evolution 2.12 and 2.21.x for all the distros supported by OpenSUSE build service (OpenSUSE, SLED, Fedora, Ubuntu, Debian, …). The debs might need a bit of tweaking in the spec file, which isn’t yet done, but might be done next week. In a general note, it is not feature complete when compared with the current Evolution Exchange provider (OWA/WebDAV). We are aware of memory built-up/issues in the preview build and it not fully tested. But it should be pretty fast. It can connect to MS Exchange 5.5, 2000, 2003 and 2007 though we have been primarily developing and testing MS Exchange 2007. It is based on OpenChange‘s libmapi-0.6 and many thanks to Julien and his team. The have been help us a lot, with the library, sample codes etc. They are working hard now libmapi-0.7 which would  give some important things for us to take the provider forward.

Unfortunately the Evolution MAPI provider won’t be shipped along with the Evolution 2.22/GNOME 2.22 release as the provider development schedule ends later than the GNOME release. So we would be releasing periodic standalone builds that can be installed on top of your Evolution/EDS installations and it should work seamlessly. We hope that various distros can pick these for their users later on. We are also thinking of nightly/weekly builds through build service, but that may take little more time, since we are now busy shaping the provider. Anyways try the preview build or build from source (EXCHANGE_MAPI_BRANCH of Evolution and Evolution-Data Server in the GNOME svn) and give your feedbacks on the wiki / mailing list as we don’t have much things for bugzilla now.

Evolution stacktrace hackfest

I just send an announcement mail to the evolution hackers list about the Evolution hackfest. We are planning to fix crashers that are marked with the ‘stacktrace’ keyword on Tuesday Jan 22nd. The keyword typically mean that the crasher has good stack trace that will be useful for the developers to fix the bug.

How can you help us?

  • If you know of bugs/crashers that are easily reproducible and aren’t part of the list, please add to it to the wiki ( http://www.go-evolution.org/Hackfest ).
  • If you have filed crashers before and have good traces/debug information, but aren’t marked with stacktrace, please add keyword then.
  • You can hack with us and fix Evolution crasher bugs. We would be on #evolution on GimpNet working on Jan 22nd all through the day.

Thanks in Advance :)

Camel DB (Disk) Summary – Evolution memory improvements/thoughts

Finally I got my chance to use some of my ITO time (just 3 days this time). I decided to spend my time towards answering memory issues of Evolution Mail. Folder summary (Message list) is one of the biggest reason for Evolution’s high memory usage. The folder summary has Message infos. Every message info is nothing but headers like from/to/cc/sent-received date etc. Notzed (One of the Ex-Evolution/Camel Hacker) wrote a design/code on addressing the core issues. But unfortunately when he left Novell, the code/design wasn’t developed. It used libdb and had lots of new design/ways to access mail data. I was thinking on the same lines and decided to take some concepts from there and wrote a new design/thought . I had spent my 3 days and nights on improving my design and I have a fully working prototype. I used sqlite as the database for Camel Summary and message UID as the primary key. All over the summary code, I used sql queries like “select * from Inbox where uid=’342′” to access the data.

I modified the entire design of Evolution/Camel to store use only UID where ever required. When ever the Message info is required it queries the DB and gets the the data and frees when not required. It could mean that we don’t need to keep the visited folders in memory for the sake of trash/junk. We don’t need to keep the vfolder’s sub folder in memory. It could mean that just the viewed folder (why this also?) can be in memory rest could be just in data base and queried as and when required. I made a prototype with this in mind and I was able to achieve what I thought. (Asking when I’m gonna commit this? Hmm, I have made a prototype. Folder Summary listing works, Junk/Trash works, Search works, VFolders works. But there are lot of things that I can/ need to optimize since I have the flexibility with DB. Since this will break ABI and add more APIs and deprecate a few, I need to design the APIs with lots of things in mind like (remote view, Mails part of EDS, etc). I was discussing with Fejj (another Mail/Camel hacker) on friday and he gave nice thoughts/inputs on my design, like having a LRU implentation to decide what message infos to keep in memory and what not (He gave the code for LRU from GMime). These optimization would reduce huge memory for users having lots of vfolders and folders. Unfortunately this may not have any effect on users lying around just one folder (Just Inbox) and huge mails in it.And after all this optimization of memory, there isn’t be any performance drop, infact, it is a bit faster now with indexed tables. But vfolders was a bit slow, but having persistent summary for vfolders, it is going to be faster than the current implementation (I haven’t prototyped this though). Achieving all this in a cleaner way will be my first target milestone and might take as close as a month. (If I’m allowed to work on this, to go full stretch on this for a month).

In current Evolution, you wont see that memory drop
Note: The drop in memory, is something you won’t see in the current Evolution. Junk/Trash keeps those last visited foders in memory.

Of course there are next levels to this. Remote view & Search-in-disk

Remote View: Currently after my first target, only the viewed folder’s message list is going to be in memory. Now we can have a custom model store, that just maps viewed message list’s infos and may be a buffer of 50/100 above and below the message lists view. We can have a cursor in the db that just moves maps the view+head/tail to memory. It means that when you have a folder of 100,000 mails and the message list shows 50 mails and the head/tail buffer is 50, you would have just 150 mail’s message info on the memory and nothing else. It may be a bit slow, if you do page down/up faster or scroll using mouse. But can make Evolution run on any machine with low memory or mobile devices (Nokia 800/810 etc… Of course you can do a optimized design to over come the performance issue with huge scroll. (Sqlite is pretty fast and possible that you may not notice it most of the times). Also it requires lots of things like sorting/threading etc needs to be built inside the tables and the cursor needs to be mapped to the message list/etree. There is no prototype/data for this, but this is possible for sure after my first milestone. This will be the second mile stone

Search-in-disk: Currently for search, the entire folder summary is brought in memory (anyways in my first milestone only this is in memory). But if we implement a remote view, it may not be so efficient to it this way. We can extend the search to be done inside the data base and just retrieve the uids or use the cursor to map the contents to the message list. Effect: It will be super fast and again on low memory consumption.

I’m not doing much for the second or the third milestone right now. But I want to work on the first milestone for Evolution 2.23.1/2 (Sorry not for GNOME 2.22, too late to bring such a huge design change) The second and the third mile stone might not bring much ABI/API changes if it is designed well in the first mile stone and can be taken/done at any point with out much disturbances to the stability IMO. I wish I w(c)ould on all this.