Camel DB (Disk) Summary – Evolution memory improvements/thoughts

Finally I got my chance to use some of my ITO time (just 3 days this time). I decided to spend my time towards answering memory issues of Evolution Mail. Folder summary (Message list) is one of the biggest reason for Evolution’s high memory usage. The folder summary has Message infos. Every message info is nothing but headers like from/to/cc/sent-received date etc. Notzed (One of the Ex-Evolution/Camel Hacker) wrote a design/code on addressing the core issues. But unfortunately when he left Novell, the code/design wasn’t developed. It used libdb and had lots of new design/ways to access mail data. I was thinking on the same lines and decided to take some concepts from there and wrote a new design/thought . I had spent my 3 days and nights on improving my design and I have a fully working prototype. I used sqlite as the database for Camel Summary and message UID as the primary key. All over the summary code, I used sql queries like “select * from Inbox where uid=’342′” to access the data.

I modified the entire design of Evolution/Camel to store use only UID where ever required. When ever the Message info is required it queries the DB and gets the the data and frees when not required. It could mean that we don’t need to keep the visited folders in memory for the sake of trash/junk. We don’t need to keep the vfolder’s sub folder in memory. It could mean that just the viewed folder (why this also?) can be in memory rest could be just in data base and queried as and when required. I made a prototype with this in mind and I was able to achieve what I thought. (Asking when I’m gonna commit this? Hmm, I have made a prototype. Folder Summary listing works, Junk/Trash works, Search works, VFolders works. But there are lot of things that I can/ need to optimize since I have the flexibility with DB. Since this will break ABI and add more APIs and deprecate a few, I need to design the APIs with lots of things in mind like (remote view, Mails part of EDS, etc). I was discussing with Fejj (another Mail/Camel hacker) on friday and he gave nice thoughts/inputs on my design, like having a LRU implentation to decide what message infos to keep in memory and what not (He gave the code for LRU from GMime). These optimization would reduce huge memory for users having lots of vfolders and folders. Unfortunately this may not have any effect on users lying around just one folder (Just Inbox) and huge mails in it.And after all this optimization of memory, there isn’t be any performance drop, infact, it is a bit faster now with indexed tables. But vfolders was a bit slow, but having persistent summary for vfolders, it is going to be faster than the current implementation (I haven’t prototyped this though). Achieving all this in a cleaner way will be my first target milestone and might take as close as a month. (If I’m allowed to work on this, to go full stretch on this for a month).

In current Evolution, you wont see that memory drop
Note: The drop in memory, is something you won’t see in the current Evolution. Junk/Trash keeps those last visited foders in memory.

Of course there are next levels to this. Remote view & Search-in-disk

Remote View: Currently after my first target, only the viewed folder’s message list is going to be in memory. Now we can have a custom model store, that just maps viewed message list’s infos and may be a buffer of 50/100 above and below the message lists view. We can have a cursor in the db that just moves maps the view+head/tail to memory. It means that when you have a folder of 100,000 mails and the message list shows 50 mails and the head/tail buffer is 50, you would have just 150 mail’s message info on the memory and nothing else. It may be a bit slow, if you do page down/up faster or scroll using mouse. But can make Evolution run on any machine with low memory or mobile devices (Nokia 800/810 etc… Of course you can do a optimized design to over come the performance issue with huge scroll. (Sqlite is pretty fast and possible that you may not notice it most of the times). Also it requires lots of things like sorting/threading etc needs to be built inside the tables and the cursor needs to be mapped to the message list/etree. There is no prototype/data for this, but this is possible for sure after my first milestone. This will be the second mile stone

Search-in-disk: Currently for search, the entire folder summary is brought in memory (anyways in my first milestone only this is in memory). But if we implement a remote view, it may not be so efficient to it this way. We can extend the search to be done inside the data base and just retrieve the uids or use the cursor to map the contents to the message list. Effect: It will be super fast and again on low memory consumption.

I’m not doing much for the second or the third milestone right now. But I want to work on the first milestone for Evolution 2.23.1/2 (Sorry not for GNOME 2.22, too late to bring such a huge design change) The second and the third mile stone might not bring much ABI/API changes if it is designed well in the first mile stone and can be taken/done at any point with out much disturbances to the stability IMO. I wish I w(c)ould on all this.

Annoying popups with Evolution? hmm…No more

Lots of people have been complaining about annoying popups with Evolution saying “Ping failed”, “Name lookup failed” etc.. I started writing a design/interface in the wiki few months back and had a test code which I showed it to a few people at Guadec 07. Since then, I was busy with too many other things and  never had  time to relook at this. Last week my wife and my son had left to my parent’s town and since then, I seem to have too much free time and during the last weekend I thought, I’ll give it for Non-Intrusive Error reporting framework and so It is part of Evolution 2.21.4 release. I’m sure that it is gonna change a lot of things and give more fine-grained control over Evolution to the user. Some of the high lights are…

  • In general every operation that the user performs is executed in a thread in a asynchronous fashion. For every such operation, there will be a status indication of that in the status bar, with a option to cancel it. It means that (Assume that you don’t have few messages cached locally) when you click a message to view, it would start fetching from the server and you realize that you wanted to click the next one and you click that too. Unfortunately till the first operation ends, the second won’t start as most backends sequence the tasks. Since each of them is represented as a task with ‘x’ – cancel button you can cancel the first one alone and so the second can pick up. This is just one simple example and I’m sure that it will be more useful in lot other scenarios as well.

task.png

  • This is about non-intrusive reporting. Every such task that goes on a asynchronous fashion, can pop up a error. Now the status bar messages are persistent and timed. Every task representation is converted to a error, if the task ended with a error. Which means “Refreshing folder…” when the error occurs would change to “Error while Refreshing folder” with a ‘!’ icon on the status bar. The user can click the icon (tool icon) to view the actual error dialog. If the user ignores it, it hides from the status bar after a specified time (configurable) with a log (Yes, Evolution features a log view along with this)

error.png

  • Evolution now features a log viewer, where all the log messages of the session are stored. The UI has provisions for Error, Warning and Debug/Info messages. But all of them aren’t yet implemented and only error logging is only done. I hope that I would do the rest in the upcoming weekends.

debuglog.png

Note: The dialog is bit outdated. It now sorts on time/descending order and has nice spacing and border and scrolled window as well.

Now the base framework is done and it is integrated with the mail thread operation, I’m hoping that I would extend my framework to most of the other operations in mailer and then in the next release I would integrate Calendar and other components.

Milan Crha extended Sankar’s work on custom labels/message tagging and the base code has hit the trunk for 2.21.4. Few more patches are still waiting and in together it is gonna make it really cool. Apart from that lots of bug fixes and some important crash/performance patches also made to 2.21.4. For a more detailed entries and release notes, see my announcement mail.

Evolution doc issues

Andre posted a mail on the hackers list about the state of Evolution docs. The doc team here at Novell uses FrameMaker to manage documentation. So the master copy that we use is frame files which can only be exported to XML and no way to import/sync from xml. So any commits that happen for the XML file on the svn needs to be manually synced. At times, some patches are XML fixes which are auto-generated by FrameMaker. In which cases, the commit from Frame files to svn would overwrite them. This is a big issue. One solution that came our is hand editing XML files for documentation. Definitely hand editing the files isn’t going to be easy (it is 8K lines file with almost 1K modification every release). It can be more prone to errors than now. I really want to know how other projects over come this?. Are there any other tools or process other projects use? Or just hand edit XML files? Any better suggestions to me? I would be happy to hear any suggestions/processes that already other projects use.

Evolution 2.22 progress updates

I just released Evolution 2.21.3 (Yes we synced Evolution with GNOME version) and send a update to the evolution hackers list. Some highlights of that. Till now, we have committed

  • Google Calendar support (Read/Write – You can create meetings)
  • External Editor support for composer – You can use Vim or Emacs for composing
  • Lots of bug fixes

I hope that I should be able to get

  • Exchange Smart card support
  • Non-intrusive error reporting
  • Matt’s new look composer to trunk
  • UAM (Unified account management) – This is the base work required to do split.

We are spending lot of effort on MAPI based Exchange connector developed in a branch (evolution and evolution-data-server). I’m not sure, if we can make it to 2.22 but surely, it can be added as a plugin later on. This connector should be able to connect to Exchange 5.5, 2000/2003/2007 servers. It is based on OpenChange’s libmapi.
I hope that things would be much clearer after a few weeks.

I don’t feel like doing anything

For the past 2 months, either me or my wife or my son was sick and down with cold and flu. My son had recurring 3-4 days of flu for almost three times over the last month and I almost took 2 week vacation to get him better. I was back from my vacation during last week. Just a few days after I returned from my vacation, his flu started again (last friday, Nov 23). Till now he was down with fever and yesterday I got him hospitalized him at Apollo. All the tests related to flu failed and the doctor concluded that it is FUO (Fever of Unidentified Origin) and then the doctor requested for an abdominal CT Scan and that too turned to be all fine and finally we did a chest x-ray to see that it is some sort of Pneumonia. I was so worried he had non-stop fever of 102+ F  for almost 8 days and no signs of coming down. Just after the test, the doctor started an anti-biotic and I’m hoping that his fever should come down in a few days. Doctor advised us to keep him at hospital for atleast 5 days. I don’t feel like doing anything 🙁

Evolution 2.12 released

I released Evolution 2.12 with GNOME 2.20. This is my first release as the maintainer for Evolution. Some interesting things we did for Evolution 2.12 follows

Spam Improvements

  • New bogofilter spam plugin
  • Spam plugins (bogofilter/spamassasin) can now learn ham
  • Spams don’t initiate new mail notifications
  • and…

Junk options

UI Improvements

  • Basic contact merging support.
  • Improved search support for Calendar and Tasks
    • New advanced search and better quick search options
  • Better printing support for mailer
  • Magic space bar support – Read all the unread mails in all folders with just space bar key
  • Message selection improvements
  • More options for offline support
  • New –disable-preview command line option to avoid repeated crash on last selected email
  • Option for Outlook style signature on top – more details here
  • Option for Outlook style reply header
  • Spinner in the status bar
  • Support for multimedia keys
  • Support for showing contact photos in preview pane
  • Thread sorting based on the date of the latest message

Performance Improvements

  • Addressbook improvements from DBus port branch
  • EContact/VCard folding optimizations
  • Huge Exchange folder loading optimizations

New Plugins

  • Bogofilter spam plugin
  • Attachment reminder plugin

Attachment reminder

  • Mail notification plugin

Mail notification

  • TNEF attachment plugins (Experimental)
  • Revamped backup/restore plugin

Restore

Lots of Exchange/GroupWise performance/stability fixes went in 2.12 release. We had a special focus on stability in 2.12 cycle and Evolution has nearly 100 crashers fixed and in total 450 bugs were fixed during the 2.12 cycle. Now we are preparing a planning page for the next cycle here. Feel free to add your thoughts to the planning page.

Thanks a lot for the contributors and we are looking forward for the next release.

Re: Camel-lite

Evolution’s goals (camel) are definitely different from Tinymail’s goal (camel-lite). You definitely can’t substitute each other in all places. Users wont be happy to use a email client that doesn’t even support message threading on a desktop. Philip posted an overview camel-lite onto the evolution-hackers list some time back. Camel-lite has a lot of things that can be merged with upstream camel like pop summary, some IMAP improvements (non-hacky stuffs). I’m sure that the discussion is ON between Varadhan and Philip to get the pop summary into trunk for early 2.14 cycle and more merging of useful things down the time. But I won’t / don’t think Evolution would move away from camel to camel-lite any time.

Update: I just saw that Sankar (who maintains EPlugins) has posted a nice blog about Evolution/Camel/Camel-lite. Though I don’t agree to all the points, some interesting differences and goals.

Morning at GUADEC

It is really nice to be at GUADEC and see the ever busy GNOME hackers

Hackers waiting before Registration

I had my talk first in the morning. Guenther saved me from my Magic Spacebar demo failure. (Caret mode was ON) It went pretty good otherwise. Unfortunately there was no recording as I have heard before.

At my talk

I had a nice chat with the audience after the talk. Nice to see the Evolution hackers from China.

Jeff Cai, Jedy Wang, Me, Varadhan and Parag