Upgrade puppet-git.lizard to Buster

It’s getting too hard to run PuppetDB on Stretch.

What we have to do:

• check release notes & upgrade doc
• check modules compatibility

This is a rough plan that might work:

• Move /var/lib/gitolite to its own (virtual) disk. Now we’ll have 3 gitolite disks.
• Clone the current puppet master vm (change its mac address) — creates new master.
• Temporarily rename sshd onion service on new VM
• Follow important part of the doc on creating new vm.
• Add it as a node to itself.
• Pin the puppetdb version.
• Upgrade to buster
• Ensure PuppetDB data goes through schema migration; run puppetdb upgrade -c /etc/puppetdb/conf.d/config.ini if needed.
• Make sure we have the pg_trgm PostgreSQL extension enabled.
• Unpin the puppetdb version and downgrade.
• Check puppetmaster logs for errors and deprecations and fix them.
• Point every agent to the new master once with --noop to identify obvious issues.
• Deal with issues by pushing fixes to new master.
• Migrate from hiera() to lookup().
• [until this point we can easily revert]
• Shutdown old vm and mark as no autostart.
• Disable Puppet agent on every system.
• Point the new system drive to the old VM.
• Merge the fixes in topic branches into master/production branches.
• Revert 73966287, 0cfcfd7f, 40111d15 and 18f135c4 in the manifests repo.
• Start puppet-git
• Push all updated submodules and the main manifests repo.
• Rename sshd onion service back to its official name on the new VM.
• Re-enable Puppet agent on every system, one after the other.
• Move new VM’s disks to the correct PV (= the same as the old VM’s disks).
• Delete temporary disks created for puppet-git-buster.

Related issues

• Related to #16927
• Blocks #13284

Original created by @intrigeri on 16460 (Redmine)