GitLab

* BUG SYMPTOM: when deleting a very large channel, the call to
  `phoneNumberRegistrar.destroy` fails due to a mysterious `Operation
  timeout`
* LIKELY CAUSE: awaiting the resolution of the call to
  `notifyMembers` (which takes over an hour for a large channel)
  likely causes the transaction initiated at the top of `destroy` to
  timeout
* FIX: launch notifications as "fire and forget" process in the
  background and add some more verbose logging to help debug if error persists

[h] dial socket pool to 5 (temporariarly)

See merge request !392

* we are about to delete a large channel, which will yield interesting
  data on message lag when it sends out a notification of the deletion
  to all subscribers
* in order to measure the effects of socket pooling, we are going to
  temporarily increase the socket pool back to 5 in order to get APMs
  from the transmission, then dial it back to 1 (which results in
  fewer crashes due to concurrency issues in signald)

Resolve "only alert after 2 consecuitve healthcheck failures"

Closes #340

See merge request !391

(since we won't get alerted until 2nd failure)

* cache healthcheck failures in a set (delete them after 2 rounds of healthchecks)
* when a channel fails healthcheck, consult set to see if it failed
  alst round, if so, alert admins, if not, wait until next round
* side-effect: use new `notifier.notifyMaintainers` helper in `sendHealthchecks`
  failure handler

* it omitted a `gauges` field, causing integration tests that called
  `metrics.setGauge` to incorrectly (and confusingly!) fail

[ni] update splash donation page with options for recurring payments

See merge request !390

[#347] Resolve "send channel redemption notifications instantly"

Closes #347

See merge request !389

* check if any incoming message is from a channel with a pending
  recycle request
* if so, redeem it (and notify admins/maintainers) right away!

side-effect:

* refactor dispatcher.dispatch to extract handling of early-returning
  and non-early returning side-efffects just before `relay`
  * tuck all logic that detects actionable state from system messages
    and possibly returns early after processing it into a
    `detectInterventions` helper (which returns an `intervetnion` func
    and returns early after calling it)
  * tuck all logic that detects and takes action but does not require
    returning early into a `detectSideEffects` function, which returns
    an array of side-effects to perform before proceeding to `relay`

* extract a `phoneNumberRegistrar.redeem`, which may be called from
  dispatcher as soon as a redemption message is received. it:
  * deletes recycle request for redeemed channel
  * immediately notifies channel admins and instance maintainers that
    channel has been redeemed (no more waiting 24 hours! :))
* refactor `recycleRequestRepository.evaluateRecycleRequests`
  * rename to `recycleRequest.getMaturerecycleRequests`
  * only retrieve and return mature recycle requests
  * don't worry about filtering out redeemed channel phone numbers
    anymore, since this will be done atomically and directly by calls
    to `#redeem` above
* simplify `phoneNumberRegistrar.processRecycleRequests`:
  * don't worry about redeemed nubmers
  * just get the mature requests, recycle their channels, and notify
    the admins that this happened

[hotfix] whoops! introduced bug that broke destroy

See merge request !388

* phoneNubmerRepository.destroy called `phoneNumber.destroy` instead
  of `phoneNumberRecord.destroy`
* this fixes it and adds a unit test

[hotfix] actually send broadcast messages in sequence

See merge request !387

* we introduced a fix a long time ago to send broadcast messages in sequence
  on the theory that (1) this would jam up signald's concurrency bugs
  less and (2) we had observed health checks fail less and transmit
  faster when sent in sequence
* however, we did not notice that `broadcastMessage` was only ever
  being called with one recipient (presumably to get per-recipient
  language choices honored) so we were never actually sending such
  messages in sequence
* @mari has a more comprehensive fix coming in #329, but until it
  lands, here is a small patch to make sure broadcast messages
  actually go out in sequence so we can measure if this makes any kind
  of difference in perf/reliability

[hotfix] notify maintainers on the diagnostics channel

See merge request !386

* we used to notify ourselves on BOOST IT, now we tend to get
  notifications on `DIAGNOSTICS`. let's be consistent about that!
* while at it: send these notifications in sequence. if a big channel
  is destroyed or recycled, we don't want signald to shit the bed
* aaaand... let's make `_notifyMany` handle either a concrete message
  or a notification key so we can use it as a helper for all of our
  notify functions

[hotifx] don't notify maintainers about eventless recycle jobs

See merge request !385

Resolve "admins can undo recycle"

Closes #215

See merge request !384

* was sending destruction notification in default lang (EN) to all recipients
* should send notification in reach recipient's lang
* to fix, provide new impl of `notifyMembersExcept`

* calls an async function repeatedly on an interval
* returns a function that, if invoked, cancels the next call of the
  async function

* the `common.destroy` function is odd
  * accepts a nullable channel (instead of just not being called)
  * has a funny return signature
  * the fact that it is a method on a channel rather than a function
    call that accepts a channel (or phone number) forces some tightly
    coupled testing patterns (that could be cleaner with pure functions
    on the db) without saving any db queries by doing so
* replace it with `channelRepository.destroy` and also create
  `phoneNumberRepository.destroy`: leaves cleaner test seams and less
  convoluted branching logcis
* clean up tests for destroy that (1) break b/c of this, (2) were
  tightly coupled and a bit hard to follow/change anyway

* get (classified) mature requests
* recycle the unredeemed channels
* notify admins of redeemed channels that their channel won't be recycled
* notify admins of all results (redemption/recycle/recycle-error)