load: reduce signalc lag to acceptable levels
context
- currently we can send 1,000 messages in a local load test in ~10 sec. we would like to reduce that to 1 sec.
- in !562 (merged), we identified the choke point as the many database calls that libsignal makes to update the sessoin table when calling
#getEncryptedMessages
- we also observed that replacing postgres with an in-memory h2 database reduced transmission time to ~2sec for 1,000 messages
- as such, in this MR, we will explore various strategies for improving throughput at persistence layer (more specifically, the many calls to the session protocol store) and pick the one that is the best trade off between improved performance and engineering overhead
implementation notes
some possible ideas....
- h2
- apache ignite on top of postgres: https://ignite.apache.org/arch/multi-tier-storage.html
- adapting postgres to act like h2 by increasing write-ahead-log size: https://blog.crunchydata.com/blog/tuning-your-postgres-database-for-high-write-loads
- using table partitioning in postgres: https://www.enterprisedb.com/postgres-tutorials/how-use-table-partitioning-scale-postgresql
- using jsonb data type: https://www.compose.com/articles/faster-operations-with-the-jsonb-data-type-in-postgresql/
- some sort of k/v store w/ writeahead log (mongo, redis, memcache cluster backed by kafka queue, etc...)
Edited by aguestuser