Skip to content

Use alternative way to select/deselect benchmark tests

This change will be reverted and that will affect soledad's benchmark test set. The problem is that:

  • the way pytest-benchmark selects tests to be run is based on the type of the fixtures a certain test uses.
  • to look at the type, the fixtures have to be instantiated.
  • we don't want to instantiate some fixtures if the tests will not be run.

There are some possibilities of workarounds:

  • modify collected tests on the fly to select/deselect based on some criteria (i.e. marks, test names or fixture names).
  • modify names of tests to contain the word "benchmark" and use the -k pytest option.
  • find a clever way to look at the fixture definition based on the argument name in the right hook spot to mark/unmark or select/deselect test functions. If we can do this, we should propose again as a modification to pytest-benchmark.