Skip to content

Resolve "swap signald for signal-cli" (#60)

aguestuser requested to merge 60-swap-signald-for-signal-cli into master

Closes #60 (closed)

This MR replaces signal-cli with signald (see repo here) in order to address memory consumption issues caused by the former and in preferences for the strictly better api and design of the latter.

Memory Profiling

By way of demonstrating the success of the refactor, here is an analysis of the memory consumption in the new design, compared with that of the old design (sparked by an analysis in #56 (closed) opened by @zig )...

with 0 channels...

CONTAINER           CPU %               MEM USAGE / LIMIT     MEM %               NET I/O             BLOCK I/O           PIDS
signald             0.15%               145.1MiB / 19.47GiB   0.73%               8.86kB / 0B         19.7MB / 73.7kB     18
signalboost         0.00%               49.96MiB / 19.47GiB   0.25%               18.9kB / 4.15kB     2.96MB / 0B         11
postgres            0.08%               22.71MiB / 19.47GiB   0.11%               12.8kB / 10.8kB     8.71MB / 246kB      7
  • baseline memory usage: 217.77 MB
  • basline processes in use: 36

with 10 channels...

CONTAINER           CPU %               MEM USAGE / LIMIT     MEM %               NET I/O             BLOCK I/O           PIDS
signald             0.17%               334.8MiB / 19.47GiB   1.68%               110kB / 93.3kB      11.6MB / 1.47MB     85
signalboost         0.00%               47.24MiB / 19.47GiB   0.24%               40.8kB / 13.3kB     209kB / 0B          11
postgres            0.00%               21.23MiB / 19.47GiB   0.11%               23.4kB / 21.6kB     2.75MB / 901kB      7
  • memory usage: 403.27 MB
  • memory usage delta: +185.5 MB
  • memory usage increase per channel: 18.5 MB
  • processes in use: 103
  • process usage delta: +67
  • processes per channel: 7

comparison with signal-cli design

  • increase per channel in old design vs. new design: 150 MB / 18.5 MB
  • -> memory savings under new design: ~10x
  • possible # channels on a 8GB RAM server, old design: 50
  • possible # channels on a 8GB RAM server, new design: 420
  • increase in channel capacity in new design: ~10x

Implementation Notes

  • remove signal-cli and replace it with signald
  • rename orchestrator -> registrar
  • remove docker service (b/c we no longer need to provision docker containers on the fly for new channels)
  • make signal its own top-level service, refactor it to talk to signald over unix sockets instead of signal-cli over dbus
  • call into signal service:
    • from registrar to welcome new publishers (note: we used to do this from dispatcher on startup)
    • from dispatcher to send messages (as before)
  • refactor dispatcher to:
    • talk to new signal service
    • read channel number from incoming messages instead of env var (previously provided by docker-composen)
    • not deal with welcoming logic, now handled by registrar (also remove welcomes table we used for that purpose)
  • refactor register routine to:
    • have a more regular interface
    • batch and rate-limit registration requests to avoid being rate-limited by signal
  • refactor messenger module to:
    • use unix sockets provided by new signal service
    • handle notification dispatch in a more modular/readable way
  • refactor registrar's channel module to:
    • add publishers and welcome them (via signal message) when it creates a channel
    • #create a channel instead of #activate one
    • no longer "initialize" anything (since this was mainly docker logic)
  • refactor registrar's phoneNumber#provisionN to:
    • no longer short-circuit all signal registrations if a twilio purchase fails
    • instead: filter numbers for those that succeed and only try to register those (returning all results, including errors)
  • refactor registrar/phoneNumber/register to push register/verify await logic into the signal service (vastly simplifying code)
  • refactor logger to log fatal errors and print stack traces
  • modify docker setup to:
    • have a separate container for signald (whereas signal-cli used to be wrapped in same container as application code and run as separate process with supervisord)
    • contain all signalboost-specific services (dispatcher and registrar) in one container
    • no longer use supervisord at all
    • use a name volume for the signald keystore
  • overhaul e2e tests to basically just test one provision number -> create channel life cycle (but comment it out b/c it costs $ and risks getting us rate limited by signal if overused)
Edited by aguestuser

Merge request reports