[#186] Resolve "queue and resend rate-limited messages"

Closes #186 (closed)

Satisfies #Behavior and #Implementation Notes specs from #186 (closed), which I am reproducing below b/c i'm a little to tired to write a proper MR report! :/

---

Context

• On channels with large memberships, we are starting to observe somewhat frequent occurences of rate-limiting by signal.
• When messages are rate-limited, signald does not attempt to resend them, it just drops them and emits an error message.
• This means that ~5% of intended recipients of some messages don't get them! Obviously really bad!

Value

• As a signalboost admin
• I want to know that every message I send to my channel will reach every subscriber
• So that I don't have to worry about who knows what, or think twice before using signalboost to send important information

Behavior

TLDR

enqueue rate-limited messages for resending with exponential backoff to ensure messages get through, but stop attempting to resend after a certain threshold of attempts have been made.

resending

GIVEN a message that has failed to send due to rate limit error for the first time

• THEN signalboost will notify admins it has experienced a rate limit error and will attempt to resend in 2 seconds
• AND THEN signalboost will attempt to resend the message 2 seconds later

exponential backoff

GIVEN a message that has been rate-limited for the Nth time

• THEN signalboost will notify admins of next attempted resend
• AND THEN signalboost will attempt to resend it in 2^N seconds

resend canceling

GIVEN a message that has been rate limited 8 times [1]

• THEN signalboost will notify admins that message will not be resent anymore
• THEN signalboost will not attempt to resend the message anymore

[1] ie: 8.5 minutes have elapsed since the first send attempt and the last resend delay was 4.25 minutes (256 sec) long

Implementation Details

• create resendInterval and maxResendInterval config vars
  • set resendInterval to 2 sec and maxResentInterval to 256 sec (2^8 sec, 4.25 min)
• create a resendQueue variable that is local to the dispatcher.run module and closed over by the (already existing) detectRateLimitedMessage function
  • resendQueue is a hashmap with keys hash(messageBody + username + recipientNumber) and values ResendableMessage
  • hash is some appropriately fast hashing algorithm. possibly sha1, possibly something simpler
  • ResendableMessage is an object with fields sdMessage: SdMessage and lastResendInterval: number
• every time that detectRateLimitedMessage detects a rate limited message from signald it:
  • takes the hash of the sender, recipient, and content of the SdMessage in the request field of the erorr message
  • tries to find an entry in the resendQueue matching that hash
• IF NO ENTRY MATCHING HASH, it:
  • enqueus the message for resending in 2 seconds
  • creates a new entry in the resendQueue using the (already taken) hash as the key, and setting the lastResendInterval field of the ResendableMessage in the value to 2
• IF ENTRY MATCHING HASH FOUND, it:
  • checks to see if the lastResendInverval is >= maxResendInterval
  • IF EXCEEDS MAX: it:
    • does not attempt to resend the message
    • deletes the corresponding entry from the resendQueue
  • IF DOES NOT EXCEED MAX, it:
    • multiplies the lastResendInterval by 2, producing newResendInterval
    • queues the failed message for resending in newResendInterval seconds (using wait(newResendIterval).then(...))
    • mutates the lastResendInterval value of the given message's entry in the resendQueue to the value of newResendInterval