This page contains information about benchmarks that are or were useful for solving issues. The issue that first motivated the creation of this page is #8885 (closed).
Calibration
Pytest has 2 modes of calibration:
-
Automatic: by default pytest-benchmark will try to run your function as many times needed to fit a 10 x
TIMER_RESOLUTIONperiod. See defaults below -
Pedantic: defaults to
rounds=1,warmup_rounds=0anditerations=1.
Configuration
These are the configuration options that tests that are automatically calibrated currently use (i.e. all except the ones that use txbenchmark_with_setup):
warmup_rounds=0iterations=1min_rounds=5-
min_time=9.5367431640625e-06(approximatelly10^{-5}second or 0.01 ms) --benchmark-calibration-precision=10
Mean time table
The following table was taken from a run of benchmarks by the CI on top of (effectivelly) the current master branch: https://0xacab.org/drebs/soledad/-/jobs/13422
The last two columns are used to estimate what would be a maximum sampling interval that makes sense for each function. The total time is estimated from the mean time times the number of rounds, and the maximum sampling interval is the maximum number that would provide good memory statistics for each test.
| module | name | mean time (ms) | rounds | function | ~total time (s) | maximum sampling interval (s) |
|---|---|---|---|---|---|---|
test_resources |
test_memory_intensive |
3843.1 | 5 | monitored_benchmark |
20 | 0.1 |
test_resources |
test_cpu_intensive |
5915.6 | 5 | monitored_benchmark |
30 | 0.1 |
test_crypto |
test_decrypt_doc_10k |
0.798 | 492 | txbenchmark |
0.5 | 0.1 |
test_crypto |
test_decrypt_doc_100k |
2.719 | 229 | txbenchmark |
0.5 | 0.1 |
test_crypto |
test_decrypt_doc_500k |
9.758 | 55 | txbenchmark |
0.5 | 0.1 |
test_crypto |
test_decrypt_doc_1M |
19.468 | 31 | txbenchmark |
0.6 | 0.1 |
test_crypto |
test_decrypt_doc_10M |
192.162 | 5 | txbenchmark |
1.2 | 0.1 |
test_crypto |
test_encrypt_doc_10k |
0.803 | 501 | txbenchmark |
0.4 | 0.1 |
test_crypto |
test_encrypt_doc_100k |
2.040 | 416 | txbenchmark |
0.8 | 0.1 |
test_crypto |
test_encrypt_doc_500k |
7.852 | 127 | txbenchmark |
1 | 0.1 |
test_crypto |
test_encrypt_doc_1M |
14.751 | 63 | txbenchmark |
0.9 | 0.1 |
test_crypto |
test_encrypt_doc_10M |
241.375 | 5 | txbenchmark |
1.2 | 0.1 |
test_crypto |
test_decrypt_raw_10k |
0.171 | 1695 | monitored_benchmark |
0.3 | 0.1 |
test_crypto |
test_decrypt_raw_100k |
0.912 | 961 | monitored_benchmark |
1 | 0.1 |
test_crypto |
test_decrypt_raw_500k |
3.937 | 249 | monitored_benchmark |
1 | 0.1 |
test_crypto |
test_decrypt_raw_1M |
7.833 | 136 | monitored_benchmark |
1 | 0.1 |
test_crypto |
test_decrypt_raw_10M |
80.620 | 10 | monitored_benchmark |
0.8 | 0.1 |
test_crypto |
test_encrypt_raw_10k |
0.185 | 916 | monitored_benchmark |
0.2 | 0.1 |
test_crypto |
test_encrypt_raw_100k |
0.860 | 914 | monitored_benchmark |
0.9 | 0.1 |
test_crypto |
test_encrypt_raw_500k |
3.946 | 254 | monitored_benchmark |
1 | 0.1 |
test_crypto |
test_encrypt_raw_1M |
7.748 | 125 | monitored_benchmark |
1 | 0.1 |
test_crypto |
test_encrypt_raw_10M |
82.109 | 12 | monitored_benchmark |
0.9 | 0.1 |
test_sync |
test_download_20_500k |
4426.7 | 4 | txbenchmark_with_setup |
18 | 1 |
test_sync |
test_download_100_100k |
4629.2 | 4 | txbenchmark_with_setup |
19 | 1 |
test_sync |
test_download_1000_10k |
10316.0 | 4 | txbenchmark_with_setup |
40 | 1 |
test_sync |
test_upload_20_500k |
4167.3 | 4 | txbenchmark_with_setup |
17 | 1 |
test_sync |
test_upload_100_100k |
4519.6 | 4 | txbenchmark_with_setup |
18 | 1 |
test_sync |
test_upload_1000_10k |
6831.7 | 4 | txbenchmark_with_setup |
28 | 1 |
test_sync |
test_nothing_to_sync |
195.7985 | 4 | txbenchmark_with_setup |
0.8 | 0.1 |
test_misc |
test_initialization |
233.5269 | 5 | monitored_benchmark |
1.2 | 0.1 |
test_sqlcipher |
test_async_create_100_100k |
1867.1 | 5 | txbenchmark |
10 | 1 |
test_sqlcipher |
test_async_create_20_500k |
2343.2 | 5 | txbenchmark |
12 | 1 |
test_sqlcipher |
test_async_create_1000_10k |
4008.3 | 5 | txbenchmark |
20 | 1 |