Add memory and cpu measurement to benchmarks and post to elasticsearch
This merge request adds cpu and memory stats for benchmark tests. Some notes:
- cpu percent is got by running
psutil.Process.cpu_percent()
two times, one before and one after the benchmark. It returns the percentage of time that the process occupied a cpu. Theoretically it can be bigger than 100% if the process occupies multiple cores, and i am not sure yet how that relates to GIL. - memory stats are got by concurrently sampling
psutil.Process.memory_percent()
during the run of the benchmark with a certain interval, and then calculating stats from the samples. Current interval is 1 second, but might need tunning for finer grained stats. Memory sampling will occupy some cpu, although possibly not much. Should we check the cpu cost of sampling memory? - cpu and memory stats are process-wide, so they include the whole tox/pytest/soledad-fixtures machinery. I am not sure if there is a finer/better way to measure these resources.
- Results are appended to the
extra_info
pytest-benchmark dictionary, which is included in the final json. - I suppose there's no problem of measuring cpu and memory for sequences of multiple rounds/iterations of the same function, as the actual mem percentage should be the same in each run (right?), and each run should take approximatelly the same amount of time (so cpu percentage should be more or less the same).
Some questions that remain:
- is there a finer way to measure the memory/cpu for the specific function being benchmarked?
- what is a good sampling interval for memory?
- is it actually not a problem measure cpu/memory for sequences of rounds/iterations as a whole?