Verify exceeded quota pipeline
I found the cdev's MX node stalled while trying to delivery mail to the e2e test user:
(...)
Nov 28 12:31:19 lynx leap-mx: [-] Waiting for the process of skipped mail to be done...
Nov 28 12:31:29 lynx leap-mx: [-] Waiting for the process of skipped mail to be done...
Nov 28 12:31:29 lynx leap-mx: [-] Waiting for the process of skipped mail to be done...
Looking at Soledad Server, it seems that it was also stalling while trying to reply new blob delivery attempts using the Incoming API. After restarting soledad-server, i saw many delivery attempts and many quota-exceeded messages like this:
Nov 28 12:42:00 giraffe soledad-server: [leap.soledad.server._blobs#error] Error 507: Quota exceeded for user: a12ae9727e9ed3511e6464128c72aaa6
Nov 28 12:42:00 giraffe soledad-server: [-] 0.0.0.0 - - [28/Nov/2017:12:41:59 +0000] "PUT /incoming/a12ae9727e9ed3511e6464128c72aaa6/8deb3487-3828-4f4c-b792-3f673aedd249 HTTP/1.1" 507 32 "-" "-"
It felt like as if some endpoint was not releasing the connection correctly (don't know if on MX or Soledad Server side), but I am not sure if that is the problem.
We should:
-
ensure that the delivered blobs to e2e are removed after the test runs. -
double check that the quota-exceeded responses are returning and releasing resources correctly (both in MX and Soledad Server sides).
Edited by drebs