soledad.client performance tuning: checkpointing
At 0a3905d9 we merged the perf-tweaks branch, which among other things enables the WAL mode, resulting in big performance gains for mail, specially during big append operations.
While appending >100 mails or so, you can notice a "glitch" in the rate of appends (copying message.... seems to stall for one or two seconds in thunderbird, for example). I believe this is due to the autocheckpoint being done in the main thread.
With the intention of moving this blocking point from the main thread, I've implemented a checkpoint recurrent task in a separete thread in the branch https://github.com/kalikaneko/soledad/tree/feature/checkpoint_to_thread
However, from my tests there does not seem to be any gain in total time (although the checkpoint stall seems to have dissapeared, but the total time suffers).
I think the cause might be that the fixed-time checkpoints leave the wal file grow too much, making reads slower than the fixed wal_autocheckpoint at 50 pages.
In general, I think we need to have a more accurate total-time measure of a copy operation that resembles the real world perceived delay. This must take into account not only the microbenchmarsk, as imaptest does, but the whole cycle of (select+append) of the copies + manual select of the destination folder and successful fetch.
What I'm doing right now is measuring wall time on a manual copy operation in thunderbird for a fixed 100 mails copy from the same dataset.
I tried also timing with offlineimap, but local->imap synchronization tries to do a SEARCH right after the APPEND, and this takes some extra time. So, it can be good to take averages over a big dataset (500 or 1000 mails), but only to compare against themselves, since the average time per message will have the SEARCH overhead added.
Other performance tweaks that might come handy at this point are:
- Turning shared cache on, and adding read_uncommited pragma.
- Playing with the page_size and cache_size pragmas.
(from redmine: created on 2014-02-24)