Pidgin tests with certificates are fragile
Setting the TailsToaster VM to using multple CPU cores (as per #6729 (closed))
caused the Pidgin tests to fail frequently when attempting to add
certificates. The problem was that the windows we look for aren’t in
focus. Unfortunately, just using vm.focus_window
won’t be enough to
solve the problem because sometimes focusing the window will sometimes
fail (#9330 (closed) — in progress).
Note that this isn’t just a theoretical problem; similar failures can also be seen on isotester1, such as:
FindFailed: can not find PidginCertificateImportFailed.png on the screen.
Line ?, in File ? (RuntimeError)
features/pidgin.feature:82:in `Then I cannot add a certificate from the
"/home/amnesia/.gnupg" directory to Pidgin'
I created the following temporary feature to test this.
@product
Feature: Chatting anonymously using Pidgin
As a Tails user
when I chat using Pidgin
I should be able to use OTR
And I should be able to persist my Pidgin configuration
And AppArmor should prevent Pidgin from doing dangerous things
And all Internet traffic should flow only through Tor
Background:
Given a computer
And I start Tails from DVD with network unplugged and I login
Then Pidgin has the expected accounts configured with random nicknames
And I start Pidgin through the GNOME menu
And I see Pidgin's account manager window
And I close Pidgin's account manager window
And I save the state so the background can be restored next scenario
Scenario: Adding a certificate to Pidgin
Then I can add a certificate from the "/home/amnesia" directory to Pidgin
Scenario: Failing to add a certificate to Pidgin
Then I cannot add a certificate from the "/home/amnesia/.gnupg" directory to Pidgin
When I close Pidgin's certificate import failure dialog
And I close Pidgin's certificate manager
I repeated the scenarios until there were 74 in total. On isotester1 I had the following results when setting the TailsToaster VM to use 3 CPUs:
Failing Scenarios:
cucumber features/certtest.feature:19 # Scenario: Adding a certificate to Pidgin
cucumber features/certtest.feature:46 # Scenario: Adding a certificate to Pidgin
cucumber features/certtest.feature:145 # Scenario: Adding a certificate to Pidgin
cucumber features/certtest.feature:163 # Scenario: Adding a certificate to Pidgin
cucumber features/certtest.feature:184 # Scenario: Failing to add a certificate to Pidgin
cucumber features/certtest.feature:190 # Scenario: Adding a certificate to Pidgin
cucumber features/certtest.feature:199 # Scenario: Adding a certificate to Pidgin
cucumber features/certtest.feature:256 # Scenario: Failing to add a certificate to Pidgin
cucumber features/certtest.feature:289 # Scenario: Adding a certificate to Pidgin
cucumber features/certtest.feature:316 # Scenario: Adding a certificate to Pidgin
74 scenarios (10 failed, 64 passed)
666 steps (10 failed, 4 skipped, 652 passed)
20m39.316s
On my T-430 the results were even more grim.
Failing Scenarios:
cucumber features/certtest.feature:37 # Scenario: Adding a certificate to Pidgin
cucumber features/certtest.feature:46 # Scenario: Adding a certificate to Pidgin
cucumber features/certtest.feature:73 # Scenario: Adding a certificate to Pidgin
cucumber features/certtest.feature:82 # Scenario: Adding a certificate to Pidgin
cucumber features/certtest.feature:91 # Scenario: Adding a certificate to Pidgin
cucumber features/certtest.feature:121 # Scenario: Failing to add a certificate to Pidgin
cucumber features/certtest.feature:127 # Scenario: Adding a certificate to Pidgin
cucumber features/certtest.feature:130 # Scenario: Failing to add a certificate to Pidgin
cucumber features/certtest.feature:136 # Scenario: Adding a certificate to Pidgin
cucumber features/certtest.feature:139 # Scenario: Failing to add a certificate to Pidgin
cucumber features/certtest.feature:148 # Scenario: Failing to add a certificate to Pidgin
cucumber features/certtest.feature:163 # Scenario: Adding a certificate to Pidgin
cucumber features/certtest.feature:172 # Scenario: Adding a certificate to Pidgin
cucumber features/certtest.feature:184 # Scenario: Failing to add a certificate to Pidgin
cucumber features/certtest.feature:190 # Scenario: Adding a certificate to Pidgin
cucumber features/certtest.feature:199 # Scenario: Adding a certificate to Pidgin
cucumber features/certtest.feature:217 # Scenario: Adding a certificate to Pidgin
cucumber features/certtest.feature:238 # Scenario: Failing to add a certificate to Pidgin
cucumber features/certtest.feature:244 # Scenario: Adding a certificate to Pidgin
cucumber features/certtest.feature:265 # Scenario: Failing to add a certificate to Pidgin
cucumber features/certtest.feature:271 # Scenario: Adding a certificate to Pidgin
cucumber features/certtest.feature:280 # Scenario: Adding a certificate to Pidgin
cucumber features/certtest.feature:283 # Scenario: Failing to add a certificate to Pidgin
cucumber features/certtest.feature:289 # Scenario: Adding a certificate to Pidgin
cucumber features/certtest.feature:292 # Scenario: Failing to add a certificate to Pidgin
cucumber features/certtest.feature:307 # Scenario: Adding a certificate to Pidgin
cucumber features/certtest.feature:328 # Scenario: Failing to add a certificate to Pidgin
cucumber features/certtest.feature:343 # Scenario: Adding a certificate to Pidgin
74 scenarios (28 failed, 46 passed)
666 steps (28 failed, 20 skipped, 618 passed)
17m12.320s
With code that I have in my workspace the robustness improves tremendously. I have yet to see it fail (but I’m trying!)
Feature Branch: kytv:test/10222-pidgin-certificates
Parent Task: #9330 (closed)
Related issues
- Related to #5330 (closed)
Original created by @kytv on 10222 (Redmine)