Unbound did not start on Ci test deploy
strange, this is probably the first time that unbound did not run after a CI test deploy.
here's the mail from greyhound:
Tested on Sat Apr 18 20:13:26 UTC 2015 on these nodes: "rewdevcouch1 rewdevcouch2 rewdevmx1 rewdevvpn1 rewdevweb1 rewdevplain1" with following versions/git commit IDs: Provider (/home/testbot/platform-test/rewire/develop/rewire): not under version control = leap command v1.7.1 (develop ea5be4ea7b6f0b269ac54655f01c7cd6dc28ece7) = leap platform v0.7 (develop f8393f28fd3d62219f372d13b10f91a0de951f57) test-2015-04-18-200001.log Running leap test on 2015-04-18-201319 Apr 18 20:13:25 = [rewdevcouch2.rewire.org] PASS: Network > Can connect to internet? Apr 18 20:13:25 = [rewdevcouch2.rewire.org] PASS: Network > Is stunnel running? Apr 18 20:13:25 = [rewdevcouch2.rewire.org] PASS: Network > Is shorewall running? Apr 18 20:13:25 = [rewdevcouch2.rewire.org] PASS: CouchDB > Are daemons running? Apr 18 20:13:25 = [rewdevcouch2.rewire.org] PASS: CouchDB > Is CouchDB running? Apr 18 20:13:25 = [rewdevcouch2.rewire.org] PASS: CouchDB > Is cluster membership ok? Apr 18 20:13:25 = [rewdevcouch2.rewire.org] PASS: CouchDB > Are configured nodes online? Apr 18 20:13:25 = [rewdevcouch2.rewire.org] PASS: CouchDB > Do ACL users exist? Apr 18 20:13:25 = [rewdevcouch2.rewire.org] PASS: CouchDB > Do required databases exist? Apr 18 20:13:25 = [rewdevcouch2.rewire.org] PASS: CouchDB > Can records be created? Apr 18 20:13:25 = [rewdevcouch2.rewire.org] 10 tests: 10 passes, 0 skips, 0 warnings, 0 failures, 0 errors Apr 18 20:13:25 = [rewdevplain1.rewire.org] PASS: Network > Can connect to internet? Apr 18 20:13:25 = [rewdevplain1.rewire.org] PASS: Network > Is stunnel running? Apr 18 20:13:25 = [rewdevplain1.rewire.org] PASS: Network > Is shorewall running? Apr 18 20:13:25 = [rewdevplain1.rewire.org] 3 tests: 3 passes, 0 skips, 0 warnings, 0 failures, 0 errors Apr 18 20:13:25 = [rewdevvpn1.rewire.org] PASS: Network > Can connect to internet? Apr 18 20:13:25 = [rewdevvpn1.rewire.org] PASS: Network > Is stunnel running? Apr 18 20:13:25 = [rewdevvpn1.rewire.org] PASS: Network > Is shorewall running? Apr 18 20:13:25 = [rewdevvpn1.rewire.org] FAIL: OpenVPN > Are daemons running? [openvpn.rb:12]: Apr 18 20:13:25 = [rewdevvpn1.rewire.org] No running process for /usr/sbin/unbound Apr 18 20:13:25 = [rewdevvpn1.rewire.org] 4 tests: 3 passes, 0 skips, 0 warnings, 1 failures, 0 errors Apr 18 20:13:25 = [rewdevcouch1.rewire.org] PASS: Network > Can connect to internet? Apr 18 20:13:25 = [rewdevcouch1.rewire.org] PASS: Network > Is stunnel running? Apr 18 20:13:25 = [rewdevcouch1.rewire.org] PASS: Network > Is shorewall running? Apr 18 20:13:25 = [rewdevcouch1.rewire.org] PASS: CouchDB > Are daemons running? Apr 18 20:13:25 = [rewdevcouch1.rewire.org] PASS: CouchDB > Is CouchDB running? Apr 18 20:13:25 = [rewdevcouch1.rewire.org] PASS: CouchDB > Is cluster membership ok? Apr 18 20:13:25 = [rewdevcouch1.rewire.org] PASS: CouchDB > Are configured nodes online? Apr 18 20:13:25 = [rewdevcouch1.rewire.org] PASS: CouchDB > Do ACL users exist? Apr 18 20:13:25 = [rewdevcouch1.rewire.org] PASS: CouchDB > Do required databases exist? Apr 18 20:13:25 = [rewdevcouch1.rewire.org] PASS: CouchDB > Can records be created? Apr 18 20:13:25 = [rewdevcouch1.rewire.org] PASS: Soledad > Is Soledad running? Apr 18 20:13:25 = [rewdevcouch1.rewire.org] 11 tests: 11 passes, 0 skips, 0 warnings, 0 failures, 0 errors Apr 18 20:13:25 = [rewdevmx1.rewire.org] PASS: Network > Can connect to internet? Apr 18 20:13:25 = [rewdevmx1.rewire.org] PASS: Network > Is stunnel running? Apr 18 20:13:25 = [rewdevmx1.rewire.org] PASS: Network > Is shorewall running? Apr 18 20:13:25 = [rewdevmx1.rewire.org] PASS: Mx > Can contact couchdb? Apr 18 20:13:25 = [rewdevmx1.rewire.org] PASS: Mx > Can contact couchdb via haproxy? Apr 18 20:13:25 = [rewdevmx1.rewire.org] PASS: Mx > Are MX daemons running? Apr 18 20:13:25 = [rewdevmx1.rewire.org] 6 tests: 6 passes, 0 skips, 0 warnings, 0 failures, 0 errors Apr 18 20:13:25 = [rewdevweb1.rewire.org] PASS: Network > Can connect to internet? Apr 18 20:13:25 = [rewdevweb1.rewire.org] PASS: Network > Is stunnel running? Apr 18 20:13:25 = [rewdevweb1.rewire.org] PASS: Network > Is shorewall running? Apr 18 20:13:25 = [rewdevweb1.rewire.org] PASS: Webapp > Can contact couchdb? Apr 18 20:13:25 = [rewdevweb1.rewire.org] PASS: Webapp > Can contact couchdb via haproxy? Apr 18 20:13:25 = [rewdevweb1.rewire.org] PASS: Webapp > Are daemons running? Apr 18 20:13:25 = [rewdevweb1.rewire.org] PASS: Webapp > Can access webapp? Apr 18 20:13:25 = [rewdevweb1.rewire.org] PASS: Webapp > Can create and authenticate and delete user via API? Apr 18 20:13:25 = [rewdevweb1.rewire.org] PASS: Webapp > Can sync Soledad? Apr 18 20:13:25 = [rewdevweb1.rewire.org] 9 tests: 9 passes, 0 skips, 0 warnings, 0 failures, 0 errors WARNING - "leap test" failed !
unbound got configured just right at the end of the deploy:
Apr 18 20:06:27 rewdevvpn1: notice: /Stage[main]/Site_shorewall::Defaults/Augeas[shorewall-init]/returns: executed successfully
Apr 18 20:06:27 rewdevvpn1: notice: /Stage[main]/Site_config::Files/File[/var/log/leap]/group: group changed 'root' to 'adm'
Apr 18 20:06:27 rewdevvpn1: notice: /Stage[main]/Site_config::Files/File[/var/log/leap]/mode: mode changed '0755' to '0750'
Apr 18 20:06:27 rewdevvpn1: notice: /Stage[main]/Site_apt::Preferences::Unbound/Apt::Preferences_snippet[unbound]/File[/etc/apt/preferences.d/unbound]/ensure: created
Apr 18 20:06:33 rewdevvpn1: notice: /Stage[main]/Unbound::Package/Package[unbound]/ensure: ensure changed 'purged' to 'present'
Apr 18 20:06:33 rewdevvpn1: notice: /Stage[main]/Unbound/Concat[/etc/unbound/unbound.conf]/File[/var/lib/puppet/concat/_etc_unbound_unbound.conf]/ensure: created
Apr 18 20:06:33 rewdevvpn1: notice: /Stage[main]/Unbound/Concat[/etc/unbound/unbound.conf]/File[/var/lib/puppet/concat/_etc_unbound_unbound.conf/fragments.concat.out]/ensure: created
Apr 18 20:06:33 rewdevvpn1: notice: /Stage[main]/Unbound/Concat[/etc/unbound/unbound.conf]/File[/var/lib/puppet/concat/_etc_unbound_unbound.conf/fragments.concat]/ensure: created
Apr 18 20:06:33 rewdevvpn1: notice: /Stage[main]/Unbound/Concat[/etc/unbound/unbound.conf]/File[/var/lib/puppet/concat/_etc_unbound_unbound.conf/fragments]/ensure: created
Apr 18 20:06:33 rewdevvpn1: notice: /Stage[main]/Site_config::Caching_resolver/Concat::Fragment[unbound glob include]/File[/var/lib/puppet/concat/_etc_unbound_unbound.conf/fragments/10_unbound glob include]/ensure: created
Apr 18 20:06:33 rewdevvpn1: notice: /Stage[main]/Site_openvpn::Resolver/File[/etc/unbound/unbound.conf.d/vpn_unlimited_tcp_resolver.conf]/ensure: defined content as '{md5}92359685aa3d320bde037215d0fd2c9d'
Apr 18 20:06:33 rewdevvpn1: notice: /Stage[main]/Unbound/Concat::Fragment[unbound server]/File[/var/lib/puppet/concat/_etc_unbound_unbound.conf/fragments/1_unbound server]/ensure: created
Apr 18 20:06:33 rewdevvpn1: notice: /Stage[main]/Unbound/Concat[/etc/unbound/unbound.conf]/Exec[concat_/etc/unbound/unbound.conf]/returns: executed successfully
Apr 18 20:06:33 rewdevvpn1: notice: /Stage[main]/Unbound/Concat[/etc/unbound/unbound.conf]/Exec[concat_/etc/unbound/unbound.conf]: Triggered 'refresh' from 4 events
Apr 18 20:06:33 rewdevvpn1: notice: /Stage[main]/Unbound/Concat[/etc/unbound/unbound.conf]/File[/etc/unbound/unbound.conf]/content: content changed '{md5}7b9cf83ef566e394b3f259ae7b0efc7d' to '{md5}a6e51e130daf8f1d816e885c17eccb74'
Apr 18 20:06:33 rewdevvpn1: notice: /Stage[main]/Site_openvpn::Resolver/File[/etc/unbound/unbound.conf.d/vpn_unlimited_udp_resolver.conf]/ensure: defined content as '{md5}0e336bffc91a9a26d3d0fa262dca8c84'
Apr 18 20:06:34 rewdevvpn1: notice: /Stage[main]/Unbound::Service/Service[unbound]: Triggered 'refresh' from 3 events
Apr 18 20:06:34 rewdevvpn1: notice: Finished catalog run in 788214.28 seconds
Apr 18 20:06:38 rewdevvpn1: APPLY COMPLETE (changes made) {user: testbot, platform: 0.7 (develop f8393), leap_cli: 1.7.1 (develop ea5be)}
this is from syslog:
root@rewdevvpn1:~# grep unbound /var/log/syslog Apr 18 20:06:32 rewdevvpn1 unbound-anchor: /var/lib/unbound/root.key does not exist Apr 18 20:06:32 rewdevvpn1 unbound-anchor: success: the anchor is ok Apr 18 20:06:32 rewdevvpn1 unbound-anchor: /var/lib/unbound/root.key has content Apr 18 20:06:32 rewdevvpn1 unbound-anchor: success: the anchor is ok Apr 18 20:06:32 rewdevvpn1 unbound: [24733:0] notice: init module 0: validator Apr 18 20:06:32 rewdevvpn1 unbound: [24733:0] notice: init module 1: iterator Apr 18 20:06:33 rewdevvpn1 unbound: [24733:0] info: start of service (unbound 1.4.22). Apr 18 20:06:33 rewdevvpn1 unbound: [24733:0] info: service stopped (unbound 1.4.22). Apr 18 20:06:33 rewdevvpn1 unbound: [24733:0] info: server stats for thread 0: 0 queries, 0 answers from cache, 0 recursions, 0 prefetch Apr 18 20:06:33 rewdevvpn1 unbound: [24733:0] info: server stats for thread 0: requestlist max 0 avg 0 exceeded 0 jostled 0 Apr 18 20:06:34 rewdevvpn1 unbound-anchor: /var/lib/unbound/root.key has content Apr 18 20:06:34 rewdevvpn1 unbound-anchor: success: the anchor is ok
and this after i restarted later manually:
Apr 19 19:12:59 rewdevvpn1 unbound: [12893:0] notice: init module 0: validator Apr 19 19:12:59 rewdevvpn1 unbound: [12893:0] notice: init module 1: iterator Apr 19 19:12:59 rewdevvpn1 unbound: [12893:0] info: start of service (unbound 1.4.22).
(from redmine: created on 2015-04-19, closed on 2015-06-23)