The "is properly stream isolated" test suite mechanism is fragile

I’ve seen this sort of things happen a few times recently:

04:20:15.971436382: Remote shell: spawning as root: while true; do   ss -taupen | grep 'users:(("whois"';   sleep 0.1; done > /tmp/ss.log
    When I monitor the network connections of whois                                # features/step_definitions/tor.rb:280
04:20:16.164827486: Remote shell: calling as amnesia: whois 'boum.org'
04:20:17.088353577: call returned: [0, "Domain Name: BOUM.ORG\nRegistry Domain ID: D103677623-LROR\nRegistrar WHOIS Server: whois.gandi.net\nRegistrar URL: http://www.gandi.net\nUpdated Date: 2018-12-22T17:12:21Z\nCreation Date: 2004-01-07T17:49:42Z\nRegistry Expiry Date: 2028-01-07T17:49:42Z\nRegistrar Registration Expiration Date:\nRegistrar: Gandi SAS\nRegistrar IANA ID: 81\nRegistrar Abuse Contact Email: abuse@support.gandi.net\nRegistrar Abuse Contact Phone: +33.170377661\nReseller:\nDomain Status: clientTransferProhibited https://icann.org/epp#clientTransferProhibited\nRegistrant Organization: Ben voui\nRegistrant State/Province:\nRegistrant Country: FR\nName Server: NS1.INVESTICI.ORG\nName Server: NS2.INVESTICI.ORG\nName Server: NS3.INVESTICI.ORG\nName Server: NS1-V6.INVESTICI.ORG\nName Server: NS2-V6.INVESTICI.ORG\nName Server: NS3-V6.INVESTICI.ORG\nDNSSEC: unsigned\nURL of the ICANN Whois Inaccuracy Complaint Form https://www.icann.org/wicf/)\n>>> Last update of WHOIS database: 2019-09-01T12:16:28Z <<<\n\nFor more information on Whois status codes, please visit https://icann.org/epp\n\nAccess to Public Interest Registry WHOIS information is provided to assist persons in determining the contents of a domain name registration record in the Public Interest Registry registry database. The data in this record is provided by Public Interest Registry for informational purposes only, and Public Interest Registry does not guarantee its accuracy. This service is intended only for query-based access. You agree that you will use this data only for lawful purposes and that, under no circumstances will you use this data to (a) allow, enable, or otherwise support the transmission by e-mail, telephone, or facsimile of mass unsolicited, commercial advertising or solicitations to entities other than the data recipient's own existing customers; or (b) enable high volume, automated, electronic processes that send queries or data to the systems of Registry Operator, a Registrar, or Afilias except as reasonably necessary to register domain names or modify existing registrations. All rights reserved. Public Interest Registry reserves the right to modify these terms at any time. By submitting this query, you agree to abide by this policy.\n\nThe Registrar of Record identified in this output may have an RDDS service that can be queried for additional information on how to contact the Registrant, Admin, or Tech contact of the queried domain name.\n", ""]
    And I query the whois directory service for "boum.org"                         # features/step_definitions/torified_misc.rb:3
    And the whois command is successful                                            # features/step_definitions/torified_misc.rb:35
04:20:17.092921766: opening file /tmp/ss.log in 'read' mode
04:20:17.125341865: read complete
    Then I see that whois is properly stream isolated                              # features/step_definitions/tor.rb:289
      Couldn't see any connection made by whois so something is wrong.
      <false> is not true. (Test::Unit::AssertionFailedError)
      features/tor_stream_isolation.feature:40:in `Then I see that whois is properly stream isolated'

The whole thing looks racy to me:

• If whois is too fast, then the ss | grep command can miss the connection. To fix this, we need a more reliable monitoring mechanism.
• Even if ss | grep logged the connection, it’s possible that our test suite reads the log file before the spawned shell had time to write the connection to the log. To fix this, we could send SIGTERM to the spawned shell and wait for it to exit, to ensure it has flushed its STDOUT to the log file.

Regarding a more robust monitoring mechanism:

• Run whois using a dedicated GID (e.g. with sg(1)) or network namespace, and log with the firewall any connection emitted by this group/namespace.
• Run whois under an AppArmor profile and so something similar with SECMARK (not sure if the AppArmor LSM tags packets with SECMARK yet, though).
• Anything else?

Meanwhile, I’m going to tag @fragile all scenarios that rely on this mechanism.

Related issues

• Related to #17163 (closed)

Original created by @intrigeri on 17013 (Redmine)