TECH TASK EPIC: integration tests
This is an epic that will take many cards to complete. Below, find specs for a Testing Framework and a bunch of Scenarios we want to use it to test.
I imagine completing this epic by:
- playing a card to make the testing framework and integrate it with gitlab CI
- playing one card each for each of the subheaders in the Scenarios section (starting with Basic Admin Flows then Basic Subscriber Flows then Advanced Features)
- declaring victory!
Testing Framework
Overview
- An orchestrator service is responsible for:
- running the e2e mocha specs and reporting their results
- re-verifying a set of 5 signal phone numbers every time the suite runs
- spinning up and communicating with the sub-services that simulate e2e use of signalboost
- For each spec, the orchestrator:
- spins up an instance of signalboost
- spins up a set of testbots with either admin, subscriber, or random person roles
- farms out a set of messages for testbots to send
- receives any inbound messages relayed from testbots
- computes whether the combination of sent messages and received messages passes the given spec
- Testbots are responsible for:
- sending any messages that the orchestrator tells them to send
- reporting any received messages back to the orchestrator
- The orchestrator and testbots communicate with each other over https
Orchestrator
- can register and verify a set of phone numbers specified in env vars
- can start signalboost and provision it with one of the registered phone numebrs
- can start testbots, provision them with one of registered numbers, assign them a port on localhost
- exposes
POST /receivedMessage
endpoint on localhost- payload contract:
{ botId: String, message: String, timestamp: DateTime, senderPhoneNumber: String}
)
- payload contract:
- hits
POST /sendMessages
endpoint on testbots (on localhost):- payload contract:
Array<{ message: String, recipientPhoneNumber}>
- as a beginning strategy for handling multimessage flows: just space each message by some interval that allows for a response to arrive
- if more complicated conditional logic is necessary, figure out how to serialize rulesets, but try to avoid doing this!
- payload contract:
- determines whether spec conditions were met by:
- collecting all incoming messages to
/receivedMessage
- analyzing whether the correct recipients received the correct messages from the correct numbers
- e.g.: if a user subscribes, sends HELP, and an admin sends a blast, does the user receive a welcome message, get a HELP listing, and get the blast from the admin? (etc...)
- collecting all incoming messages to
Testbots
- are provisioned by orchestrator and assigned a port
- spin up an http server listening on that port
- wrap an instance of the
signal
service module (so they can send messages) - listen to incoming message directives on
POST /sendMessages
and send the messages spec'ed in the paylod to the numbers spec'ed in the same - relays any incoming messages to
POST /receivedMessage
on orchestrator
Scenarios
This is a manifest of all known scenarios that should be tested in order to complete the epic. It will likely change! Each list element will get its own card. When the card is completeed, the list element's checkbox can be completed. When all checkboxes are checked, the epic is done!
Basic Admin Flows
[ ] creating a channel
- when a random user sends a message to the signup channel, they get a response
[ ] using a new channel
- when a channel is created with admins X and Y, both X and Y get welcome messages.
- when X sends a broadcast message, both X and Y receive it (and vice versa)
- when X or Y sends HELP or INFO, they receive correct responses
[ ] adding an admin
given a channel with admins X & Y and random person Z:
- when X sends a message, Y receives it but Z does not
- if Z tries to send INFO, they get an errror message
- when X sends "ADD ", Z gets a welcome message and X and Y get notified that Z is now an admin
- when Z tries to send INFO, they responses (w/ correct number of admins)
- when X sends a message Z receives it
- when Z sends a message, X & Y receive it
[ ] removing an admin
given channel w/ admins X, Y, & Z:
- when admin X sends "REMOVE " then Y and Z receive notifications
- when Z tries to send INFO they get an error
- when Z tries to send a broadcast message, they get an error
- when X sends a broadcast message, Y receives it by Z does not
[ ] leaving
given channel w/ admins X & Y:
- when admin X sends "GOODBYE" then X receives a response & Y receives a notification.
- when X sends INFO they get an error
- when X sends a broadcast message they get an error
- when Y sends a broadcast message, X does not receive it
Basic Subscriber Flows
[ ] joining
given channel w/ admin X and random people Y and Z:
- when X sends a broadcast message, Y and Z do not receive it
- when Y sends HELLO to channel, Y receives a response and (if implemented) X receives a notification, then...
- when Y sends INFO to the channel they receive response (with correct number of subscribers)
- when X sends a broadcast message, Y sees it but Z does not
[ ] leaving
given channel w/ admin X and subscribers Y and Z
- when Y sends "GOODBYE", Y receives a response and (if implemented) X receives a notification
- when Y sends INFO they get an error
- when X sends a broadcast message, Y does not receive it
Advanced Features
[ ] hotline mode
given channel with admins X and Y, subscriber Z, and random person A; and RESPONSES OFF
- when Z or A tries to send a message, they get an error
- when X sends "RESPONSES ON" they get a success message and (if implemented) Y gets a notification [hm.. Z too?]
- when A sends a message, X and Y get it, but Z does not; A receives a successful forward response
- when X sends "RESPONSES OFF", X gets a notification [hmm... new feature: Z too?]
- when Z or A tries to send a message they get an error
[ ] multi-language support
given a channel with admin X and random people Y, Z, and A
- when Y joins with HOLA, they receive responses to INFO in spanish
- when Y sends ENGLISH, they receive responses to INFO in english
- when Z joins with HELLO, they receive responses to INFO in english
- when Z sends ESPAÑOL they receive responses to INFO in spanish
- when X adds A as an admin with "AGGREGAR", Z receives a welcome message in spanish
[ ] vouched mode (to-be implemented)
given channel with admin X and random people Y, Z, A, and B; with vouching mode ON:
- when Y sends HELLO, they get an errror
- when X sends INVITE Y receives an invite message
- when Y sends "YES", then Y receives a welcome message
- when X sends a broadcast message Y receives it
- (repeat for X inviting Z)
- when Y sends INVITE , Y receives a response, A receives nothing
- when Z also sends INVITE , Z receives a response, Y receives a notification, A receives a welcome message
- when A sends YES, A receives a welcome message
- when Z and A invite B, B receives a welcome
- when B replies NO, B receives a response, Z and A receive nothing
- when X sends a broadcast message, B does not receive it
[ ] subscriber key rollover
given channel with admin X and subscriber Y
- when Y changes safety number then X sends a broadcast message, Y receives it
[ ] admin key rollover
given channel with admins X, Y, and Z
- when X changes safety number, then Y and Z receive a notification (that X has been removed)
- when X tries to send a message, Y and Z do not receive it
- when X sends INFO, they do not get a response
- when Y sends ADD , X gets a welcome message, Z receives a notification (that X has been added)
- when X sends a message, Y and Z receive it
- when X sends INFO, they get a response