Graph soledad benchmarks with Kibana

Current WIP Dashboard Documentation

Todo

• Install elasticsearch, logstash and kibana on moose
• Enable elasticsearch auth so it can get reached from outside
• Manually puting real metrics into elasticsearch
• Fix commit_info.project and commit_info.branch
• Include missing commit_info.id
• Add commit_info.datetime to pytest-benchmark output
• Fix couchdb access in docker benchmark tests
• Make sure we dont leak elasticsearch credentials from the CI
• Merge soledad pull req to benchmark and push to elasticsearch: soledad!74 (merged) (@drebs)
  • Integrate pushing the metrics from CI builds
• Puppetize
  • /etc/elasticsearch/moose/elasticsearch.yml
  • /etc/kibana/kibana.yml
• Set concurrent=1 in quelea:/etc/gitlab-runner/config.toml
• Add hostname to json output of benchmarks
• Make sure commit_info.branch contains the correct branch, because gitlab-ci checks out a detatched HEAD by default
• Database locks in benchmark tests on quelea docker runner
• Finish one graph with all metrics for one test we want to show (min, max, mean, stdeev etc.)
• Use the above template for creating graphs for each (?) test
• Use upstream pytest-benchmark in tox.ini once all pending pull reqs are merged:
  • Add commit_info.author_time: https://github.com/ionelmc/pytest-benchmark/pull/75
  • Fix update hooks specs: https://github.com/ionelmc/pytest-benchmark/pull/74
• Two dashboards: one for historic benchmarks (run on weasel/quelea) and one for every new commit on beluga
• Decrease standard deviation by a) increasing the benchmark rounds for each test or/and b) re-running all tests multiple times again
• improve the stdev plotting: mean +/- 3stdev
• Test SSDs+HDs on quelea
• Improve kibana installation
  • Use auth_key_sha256 for passwords in /etc/elasticsearch/moose/elasticsearch.yml
  • Secure /etc/kibana/kibana.yml
  • Puppetize Installation of elasticsearch-readonlyrest-plugin
• Integrate graphs into gitlab

Gather questions for elastic call

• Add summary to each graph that explains what's measured
• Add time unit to y-axis
• Group commits per week ?
• Saved search to refine an existing search