Skip to content

Fix newly identified issues to make our test suite more robust and faster

Our initial plan will be to mark any scenario that we ever see fail in the automated test suite run by jenkins (not locally, or anywhere else) with the @fragile tag. On jenkins we will run the test suite with the Cucumber option --tag ~fragile@, which makes it skip these scenarios.

Whenever we find a robustness issue for a Scenario $SCENARIO we do the following:

  1. Add the @fragile tag to $SCENARIO, commit it to a suitable base branch B. Often B will be stable, but it could be devel if not affecting stable, or e.g. feature/jessie (or other long-term integration branches) if only affecting that one. Let’s say this became commit DEADBEE. Then merge B into all base branches where it makes sense (e.g. if B == stable we’d merge into devel, and possible then merge devel into feature/jessie).
  2. File a ticket (let’s say the number becomes #NNNNN) with subject: “$SCENARIO is fragile” (or similar) and
    1. reference commit DEADBEE in the description.
    2. make it block this ticket, i.e. #10288.
  3. Create a branch test/NNNN-fix-${SCENARIO} (or a similar, shortened name) from B, and
    1. commit a revert of DEADBEE.
    2. set this branch as the Feature Branch in ticket #NNNN.

Iterating this process (and merging the base branches into all feature/bugfix/test branches, including those created by this procedure) should make us converge to a state where we have isolated all robustness issues to individual branches, and all base branches should be green. On jenkins.

Note: There may be more than one reason to tag a scenario @fragile, which breaks the above scheme a bit. We do not want to end up with the revert of one branch removing the @fragile tag in the base branches when its merged, while there still are at least one unmerged branch with another reason for the scenario to be marked @fragile. I think the best way to track this is on the ticket, and by making a comment around the @fragile tag, listing each ticket tracking the scenarios fragility. In each branch you remove only its ticket => we get a merge conflict as a “notification” when merging, and we only remove the tag in the base branch when the last ticket is removed from the comment.

Creating a summary of failures

1. Clone the puppet-tails Git repo, get the attached json-analysis script.

2. Get the `jobResults-*.xml` files you want from jenkins.lizard:/var/lib/jenkins/global-build-stats/jobresults/

3. download all the JSON test result files you’re interested in (you can pass an epoch to ISO-test-suite-runs, that’ll be the starting point) e.g.:

cd $PUPPET_TAILS_REPO
for url in $(./files/jenkins/master/ISO-test-suite-runs /tmp/jobResults-2015-12.xml) ; do
    dest=$(mktemp --tmpdir=. tailstester-XXXXXXXXXX.json)
    wget -O "$dest" "$url"
    [ -s "$dest" ] || rm -f "$dest"
done

4. Do the analysis on JSON files:

json-analysis --steps *.json@

Attachments

Subtasks

Related issues

Original created by @anonym on 10288 (Redmine)

To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information