[#290] Resolve "add basic APM monitoring to signald"

Closes #290 (closed)

context

• we want to start investigating message delivery lag under load by instrumenting signald with prometheus metrics exporting capabilites
• the goal of this MR is to get the ball rolling by measuring the simplest valuable thing: what proportion of time in Manager.sendMessage is spent in network calls vs. file system operations

implementation details

use signald build with prometheus instrumentation:

• repo: https://0xacab.org/team-friendo/signald-fork.git
• branch: master
• commit: 0ece4970980a5f4d768339fbe067baa6cc6f62a0

expose signald metrics exporting server from same box as signalboost app by exttracting signalboost/signald host/port into env vars and injecting them into docker configs

• context: we want to provide different VIRTUAL_HOST and VIRTUAL_PORT env vars to signalboost and signald containers so that our ngix container can proxy traffic to each app container correctly based on hostname and port on incoming requests
• solution: declare SIGNALBOST_HOST_URL, SIGNALBOOST_PORT, SIGNALD_HOST_URL, and SIGNALS_PORT in .env, read them into the appropriate containers in docker-compose files

restrict access to signald metrics server by whitelisting prometheus server IP in signald ngix configs

• context: we want to expose an http server from the signald container without doing too much authentication work (in part b/c the built in server doesn't support inspecting headers, etc. and we don't want to bother writing a jetty server just for this one endpoitn)
• solution:
  • expose an unauthenticated endpoint (only accessed over TLS) in the signald container and map traffic from signald.signalboost.info to this container via the nginx proxy container
  • whitelist localhost and the prometheus server's IP address as the only address that can access signald.signalboost.info via the vhost.d file for the signald.signalboost.info config file injected into our nginx proxy
• credentials book-keeping:
  • registered in blackbox: nginx/vhost.d/signald.signalboost.info (to accomplish IP whitelisting)
  • edited in blackbox: prometheus/prometheus_dev.yml, prometheus_prod.yml (to add signald scrape job configs)

added Doing devops perf labels

changed title from Resolve "add basic APM monitoring to signald" to [#290] Resolve "add basic APM monitoring to signald"

enabled an automatic merge when the pipeline for d3d7aadb succeeds

canceled the automatic merge

added 2 commits

• c6dea470 - [290] configure prometheus to scrape signald in prod
• 49ebba4a - [290] strip auth token from prometheus dev signald scrape configs

Compare with previous version

changed the description

mentioned in commit 7300614a

merged

mentioned in commit e7e16b75

mentioned in merge request !325 (merged)

mentioned in commit f611d14c

mentioned in commit eb162542

mentioned in merge request !327 (merged)

mentioned in commit 9ae1b420