load: identify choke point in signalc send path

context

  • with #201 (closed), we demonstrated that signalc can handle sending 1,000 messages with significantly higher throughput than signald
  • however, it still takes roughly 20sec to transmit all 1,000 messages (vs. an acceptable 2 sec for 100 messages)
  • in this card, we would like to identify the point(s) in the execution flow

implementation details

  • map out the execution of handling a send request from signalboost into stages (from receiving the message off the wire, to sending a response back) in intuitive language that helps us reason about throughput and flow (do "domain modeling" for our application's performance)
  • measure the time spent waiting in queue at each stage in this flow, find the part of the flow where messages stack up (and spend a long time waiting in the queue) -> this is the chokepoint
  • optionally: corroborate the above measurement by measuring throughput at various phases of the flow (with the caveat that throughput will go down throughout the flow once a backup has started at the chokepoint --> on its own this measure will not help us identify the chokepoint, but will perhaps be useful for helping identify when it begins)

out of scope

  • clearing the choke point! that is left for #459
Edited by aguestuser