sc: handle restarts gracefully

behavior

WHEN docker restarts signalc (for a db backup, deploy, or after a healthcheck failure)

• THEN signalc will restart without dropping any messages that were enqueued to be received or sent at the time of shutdown
• AND THEN signalc will responsively indicate to callers once it has restarted by responding to is_alive checks (only happens in healthcheck failure)

implementation notes

• sc send path:

  • each call to SignalSender#send:
    • increments messagesInFlight (atomic int) before calling libsignal
    • decrements when sendResult returned

• app shutdown hook:

  • Application#init registers a listener for SIGTERM (as per https://stackoverflow.com/questions/2541597/how-to-gracefully-handle-the-sigkill-signal-in-java)
  • polls app.signalSender.messagesInFlight until it reaches 0 (or a 1 min-ish timeout)
  • if timeout exceeded, notifies prometheus and waits nother 5 sec (for counter to be scraped)

• receive path:

  • write envelope to db before decrypt
  • delete envelope to db after decrypt (and socket send to sb)
  • on restart in subscribe, clear queue before opening message pipe for new messages

• docker:

  • modify shutdown for signalc container to only send SIGTERM or to wait longer before sending SIGKILL
  • as per: https://www.ctl.io/developers/blog/post/gracefully-stopping-docker-containers/

• liveness check:

  • modify signald to respond to {type: is_alive} by echoing it back
  • implement handler in signalc to do the same, ignore version messages!
  • modify signalboost to use the new isAlive:
    • don't reason about versions
    • use id in response to trigger callback