Skip to content

GitLab

Last edited by drebs
Page history

benchmarks

This page contains information about benchmarks that are or were useful for solving issues. The issue that first motivated the creation of this page is #8885 (closed).

Calibration

Pytest has 2 modes of calibration:

  • Automatic: by default pytest-benchmark will try to run your function as many times needed to fit a 10 x TIMER_RESOLUTION period. See defaults below
  • Pedantic: defaults to rounds=1, warmup_rounds=0 and iterations=1.

Configuration

These are the configuration options that tests that are automatically calibrated currently use (i.e. all except the ones that use txbenchmark_with_setup):

  • warmup_rounds=0
  • iterations=1
  • min_rounds=5
  • min_time=9.5367431640625e-06 (approximatelly 10^{-5} second or 0.01 ms)
  • --benchmark-calibration-precision=10

Mean time table

The following table was taken from a run of benchmarks by the CI on top of (effectivelly) the current master branch: https://0xacab.org/drebs/soledad/-/jobs/13422

The last two columns are used to estimate what would be a maximum sampling interval that makes sense for each function. The total time is estimated from the mean time times the number of rounds, and the maximum sampling interval is the maximum number that would provide good memory statistics for each test.

module name mean time (ms) rounds function ~total time (s) maximum sampling interval (s)
test_resources test_memory_intensive 3843.1 5 monitored_benchmark 20 1
test_resources test_cpu_intensive 5915.6 5 monitored_benchmark 30 1
test_crypto test_decrypt_doc_10k 0.798 492 txbenchmark 0.5 0.1
test_crypto test_decrypt_doc_100k 2.719 229 txbenchmark 0.5 0.1
test_crypto test_decrypt_doc_500k 9.758 55 txbenchmark 0.5 0.1
test_crypto test_decrypt_doc_1M 19.468 31 txbenchmark 0.6 0.1
test_crypto test_decrypt_doc_10M 192.162 5 txbenchmark 1.2 0.1
test_crypto test_encrypt_doc_10k 0.803 501 txbenchmark 0.4 0.1
test_crypto test_encrypt_doc_100k 2.040 416 txbenchmark 0.8 0.1
test_crypto test_encrypt_doc_500k 7.852 127 txbenchmark 1 0.1
test_crypto test_encrypt_doc_1M 14.751 63 txbenchmark 0.9 0.1
test_crypto test_encrypt_doc_10M 241.375 5 txbenchmark 1.2 0.1
test_crypto test_decrypt_raw_10k 0.171 1695 monitored_benchmark 0.3 0.1
test_crypto test_decrypt_raw_100k 0.912 961 monitored_benchmark 1 0.1
test_crypto test_decrypt_raw_500k 3.937 249 monitored_benchmark 1 0.1
test_crypto test_decrypt_raw_1M 7.833 136 monitored_benchmark 1 0.1
test_crypto test_decrypt_raw_10M 80.620 10 monitored_benchmark 0.8 0.1
test_crypto test_encrypt_raw_10k 0.185 916 monitored_benchmark 0.2 0.1
test_crypto test_encrypt_raw_100k 0.860 914 monitored_benchmark 0.9 0.1
test_crypto test_encrypt_raw_500k 3.946 254 monitored_benchmark 1 0.1
test_crypto test_encrypt_raw_1M 7.748 125 monitored_benchmark 1 0.1
test_crypto test_encrypt_raw_10M 82.109 12 monitored_benchmark 0.9 0.1
test_sync test_download_20_500k 4426.7 4 txbenchmark_with_setup 18 1
test_sync test_download_100_100k 4629.2 4 txbenchmark_with_setup 19 1
test_sync test_download_1000_10k 10316.0 4 txbenchmark_with_setup 40 1
test_sync test_upload_20_500k 4167.3 4 txbenchmark_with_setup 17 1
test_sync test_upload_100_100k 4519.6 4 txbenchmark_with_setup 18 1
test_sync test_upload_1000_10k 6831.7 4 txbenchmark_with_setup 28 1
test_sync test_nothing_to_sync 195.7985 4 txbenchmark_with_setup 0.8 0.1
test_misc test_initialization 233.5269 5 monitored_benchmark 1.2 0.1
test_sqlcipher test_async_create_100_100k 1867.1 5 txbenchmark 10 1
test_sqlcipher test_async_create_20_500k 2343.2 5 txbenchmark 12 1
test_sqlcipher test_async_create_1000_10k 4008.3 5 txbenchmark 20 1

Back in history

The following information should be documented in a canonical place so devs will always know what is going on:

  • We use the machine called weasel for running soledad benchmarks. It is a VM dedicated to this.
  • We current display benchmark results in https://benchmarks.leap.se/