design a way to benchmark server scalability

changed milestone to %Need triaging

changed milestone to %0.10.1

changed milestone to %0.10.2

changed milestone to %Cycle 6

assigned to @drebs

If we define "scalability" as "the ability to serve X requests in a certain amount of time while maintaining low usage levels of memory/cpu", then we have to define:

what is the expected number of requests.
what is a low usage level of memory/cpu.
what is the request timeout for satisfying this.

We can start by running benchmark tests and observing what is the current situation. I propose we start with the following test:

spawn a server.
watch resources consumed by server.
launch 1/10/100/1000 clients in parallel
run different tasks:
- nothing to sync.
- sync of, say, 10MB of JSON.
- sync of, say, 10MB of blobs.
measure:
- time taken to answer all requests.
- cpu usage.
- max and mean memory usages.

Does that make sense as an initial step? After that, we could iterate and improve the tests and the output to come up with more data, if needed.

@drebs your overall approach sounds right. however, I don't think we really need to start by setting an expected number of requests, or even defining what a "low" usage is, but rather focus on measuring the current status, as in what is the shape of the curve (linear? superlineal?) and what's the saturation point of the server when adding concurrent requests.

it boils down to answering the opposite questions I think: with n Gb of RAM and this many CPUs (one, hehe), what is the maximum number of clients the server can serve without timing out or crashing? From there, and knowing what's the slope of the response-to-concurrency curve, I think we have enough to start profiling better and measuring improvements.

a couple of more things:

I think we'd need more steps in the number of clients, to have a nice scalability curve.
I see important to isolate server and client in different machines.
Define clearly what's the "failure point" (I would guess a % of timed out requests beyond a certain cutoff, assuming a fixed standard timeout harcoded in the clients that we also control). we could investigate how other projects are dealing with this).

steps I can think of to test client/server in different machines:

decouple twisted server from tls handling.
get nginx/apache listening on some external port, and proxying to other local port where the twisted code runs in localhost. this makes easier the deployment from a virtualenv, you just need to bind to any local port (no need for root, no need to acess the global tls key material)
get a deploy-from-git running there (can be modified from current soledad deploy-from-git script).
add post-commit hooks to deploy each branch on a server X.
add a post-commit hook to run stress tests on server Y (this can be the docker runner maybe).

Regarding the structure of the test, I think something like this would do the trick:

.-------------------.   1. setup test environment in server   .----------------.
|  Test Controller  |---------------------------------------->| Soledad Server |
'-------------------'                                         '----------------'
    | | |        | |                                              ^     ^ ^ ^
    | | |        | |                                              |     | | |
    | | |        | |    2. start resource monitor                 |     | | |
    | | |        | |                   .------------------.       |     | | |
    | | |        | '------------------>| Resource Monitor |-------'     | | |
    | | |        |                     '------------------'             | | |
    | | |        |                                                      | | |
    | | |        |      3. trigger massive requester                    | | |
    | | |        '------------.                                         | | |
    | | |                     |                                         | | |
    | | |                     v                                         | | |
    | | |           .-------------------.                               | | |
    | | |           | Massive Requester |                               | | |
    | | |           '-------------------'                               | | |
    | | |                     |                                         | | |
    | | |                     v                                         | | |
    | | |       ...    ...   ...   ...  ...                             | | |
    | | | .----------------.      .----------------.                    | | |
    | | | | S.----------------..----------------.t |--------------------' | |
    | | | '--| So.---------.----------------.nt |-------------------------' |
    | | |    '---| Sol.----------------.ent |-------------------------------'
    | | |        '----| Soledad Client |----'    N - number of parallel requests
    | | |             '----------------'         M - total number of requests   
    | | |
    | | |
    | | |
    | | '-------------> 4. stop resource monitor, get results
    | |
    | '---------------> 5. cleanup environment in server
    |
    '-----------------> 6. save results

Some implementation possibilities I thought about while thinking about the above:

The massive requester can be done by using ab with the appropriate headers and bodies of requests.
Results can be saved in kibana, in an appropriate index.
The time would be kept by the test controller, while memory/cpu would be tracked by the resource monitor.

So this is a list of tasks to get something like the above running:

Server-side:
- Create a server-setup script that (re)creates a number of test databases.
- Create a resource monitor that can start/stop and returns cpu/mem/responsiveness information for a process.
- Create a test controller server that listens on a port and can trigger the two above over HTTP.
- Add a layer of TLS with nginx, disable TLS in Soledad Server.
Client-side:
- Create templates of request bodies for different sizes of blobs.
- Create a script that takes N, M, the number of users and a file with the request body and calls ab against the server.
- Create a test controller client that can trigger the two above.
- Add ability to query the test controller server.
- Define test scenarios (N, M, size of requests, number of different users, etc).
- Add a method that orchestrates the test for a scenario, run for different scenarios.
- Make the test run with enough steps in the number of clients to create a nice curve.
- Research parameters to consider tests as passing or failing.
- Run increasingly heavier scenarios until failure.
- Run different times for different metrics (cpu-time vs mem vs responsiveness)
- Add ability to save results.
Infrastructure:
- Find servers to run test (one for client and the other for server).
- Configure one server as test server.
- Configure the other server as test client.
- Configure a trigger of the test for every commit/merge.
- Come up with a nice way to present results: max clients, time/cpu/mem taken, responsiveness.
- Add graphs that show the nice results.
- Add a dashboard page for viewing all results.

@drebs I think this is a good plan, for me we can close this task and point here in the implementation issue. re. servers, I think we really don't need to run this on every commit. it makes more sense to trigger a benchmark between two releases, or between a branch and master if we're working on something perf-related.

in that sense, I think we can use cdev as a server-side, and something with good bandwith for the DDoS...

mentioned in issue #8979 (closed)

Closed in favour of: #8979 (closed)

closed

design a way to benchmark server scalability

definition of done

Designs

Child items 0

Activity