Self-host our website

New design

ikiwiki and web hosting

ikiwiki would run on www.lizard which also serves the generated content and runs ikiwiki.cgi (until #9174 is done):

• ikiwiki on www.lizard pushes changes back to the master/central tails.git repo (PO files updates, changes done in the web interface aka. ikiwiki.cgi). That’s exactly what the current production setup does.
• Ensure logging policy is OK:
  • nginx’ own logs: no IPs: OK
  • Journal, if relevant: OK, I’ve not found anything nginx-related in the Journal, even after hitting a 404
• There’s currently a cronjob that extracts connection stats from access.log (Tails boots, downloads of the detached ISO signature) and emails them to tails@b.o. We need to import that too.
• The current setup is heavily based on Apache features, while we usually run nginx on our infra:
  • .htaccess has:
    • rewrite rules: convert with http://winginx.com/en/htaccess, manage with Puppet, and then sysadmins update the rewrite rules upon request by tech writers
    • per-location HTTP header controls: add_header Cache-Control must-revalidate and expires 2h should do the job
  • language negotiation for the PO plugin; see below for a list of options and a working PoC
  • running ikiwiki.cgi: supported using fcgiwrap, see also https://serverfault.com/questions/93090/installing-ikiwiki-on-nginx-fastcgi-fcgi-wrapper and https://www.linode.com/docs/websites/wikis/ikiwiki-on-arch-linux/

language negotiation

I got language negotiation to work fine with this included in the http {} block:

map $http_accept_language $lang {
    default en;
    ~*^de de;
    ~*^fa fa;
    ~*^es es;
    ~*^fr fr;        
    ~*^it it;
    ~*^pt pt;
}

… and this included in the vhost:

location / {
    try_files $uri $uri/index.$lang.html $uri/index.en.html $uri/index.html =404;
}

Our initial options were:

• run Apache behind nginx → if we can avoid it, let’s do.

• proxy pass to a Perl handler that redirects to the best page: http://www.guido-flohr.net/simple-content-negotiation-for-nginx/

• nginx accept language module → not in Debian :/ ITP

• crude (but probably good enough) parsing of $http_accept_language: https://www.nginx.com/resources/wiki/modules/accept_language/ (“Alternative” section), https://stackoverflow.com/questions/3657614/how-to-rewrite-location-in-nginx-depending-on-the-client-browsers-language,

• Lua script: https://stackoverflow.com/a/25137080

Master/central tails.git repo

The master/central tails.git repo would move to git.puppet.t.b.o (sic) i.e. hosted on lizard’s gitolite:

• we’ll be able to use gitolite to manage ACLs which is better than the current setup at b.o, e.g. we can manage users & keys ourselves, give access to some more people to a subset of branches (e.g. we could allow some developers to push to feature/* and bugfix/* so their stuff is built and tested in Jenkins, but they would not be allowed to push to protected branches such as master, stable, testing and devel)
• maybe we can use a better CNAME that does not say “puppet”; but anyway, unless using the Tor onion service, a SSH config is needed because that Gitolite runs behind a non-standard port, so contributors with push access can as well use the onion service and we don’t bother about DNS names
• Git validation hook (file size, obsolete rewritten history, etc.) copied from the one that’s set up at b.o
• Git hook that triggers an ikiwiki update with the pingee plugin
• post-update hook that pushes to all mirrors (copied from b.o)

Migration plan

migrate the master/central tails.git repo

1. copy everything over to lizard and have gitolite push updates to all mirrors (see design above); mirrors, Git validation hook
2. have b.o disable their Git hook that pushes updates to mirrors, except lizard
3. migrate the other repos hosted at boum.org (mirror-pool-dispatcher, mirror-pool, promotion-material, uui-binary) and push to boum.org and immerda on update
4. announce downtime to Git committers
5. forbid Git committers write access to the former master/central tails.git
6. drop the ACL that allows the former repo to push to the new one
7. have b.o disable their Git hook that pushes updates to lizard
8. have the new master/central tails.git push to b.o on updates so that our website, still running at b.o at this point, is updated
9. update contribute/git doc
10. tell Git committers they can push again and ask them to update their config

bugfixes

1. fix updating master Git repo upon PO files updates / edit on the live website

prepare the new web hosting setup

1. set up the basic web server (including LE) and ikiwiki stuff on www.lizard under some temporary vhost name
2. CGI: tweak fcgiwrap.socket so that only the www-data user has access to the socket
3. have the master/central repo trigger an ikiwiki update (on Git push) on www.lizard as well, using the pingee plugin => we can test how it behaves
4. deal with the other repos needed to build our website (mirror-pool-dispatcher, mirror-pool, promotion-material, uui-binary): give them each a post-update hook that triggers a git pull in a non-bare clone on www.lizard and then calls the pingee plugin
5. import relevant stuff from the Apache vhost at b.o (e.g. HSTS)
6. try to set Content-Language header correctly (not needed since we specify this in our HTML)
7. logging
8. htaccess
9. avoid HTTP cache issues such as #16049 (closed)
10. see design above for more things to do
11. ensure there’s enough space in www.lizard:/var/log/nginx to host the amount of logs we need to keep around (even without hosting our website, at some point that directory took too much space: #12425 (closed))
12. set up rss2email
13. prevent ikiwiki.cgi from recording IP addresses in commit messages

migrate to the new web hosting setup

Once happy with the new hosting setup:

1. allow ikiwiki on www.lizard to push to the canonical repo’s master branch
2. rename/copy it to support tails.b.o, adjust ikiwiki.setup accordingly, rebuild
3. copy X.509 cert+key from the old website to the new one
4. point tails.b.o in /etc/hosts to lizard on a test machine and ensure the website hosted there works as expected
5. point DNS to lizard
6. update /etc/hosts:
   1. on www: commit e4b20b4
   2. on other systems: look for jenkins.tails.boum.org in manifests/classes.pp
7. update Git hooks: s/new\.tails\.boum\.org/tails\.boum\.org/ (or maybe just use www.lizard)
8. drop/update everything (e.g. hooks) about new.tails.b.o
9. ensure Let’s Encrypt certificate renewal has a chance to work

And once we’re convinced the new hosting setup works well enough:

1. cronjob that emails stats from logs
2. have b.o disable their rss2email feed, then set the email recipient for ours to the production one
3. replace content of .htaccess with a pointer to https://git-tails.immerda.ch/puppet-tails/tree/templates/website/nginx/rewrite_rules.conf.erb and notify the translation platform team (who runs a staging version of our website on Apache)
4. drop the temporary vhost and its files (whose name include news.tails.boum.org)
5. drop permissions from b.o to all Git repos on lizard and immerda
6. ask b.o to delete our website, Git repo, cronjob, and the corresponding SSH keys
7. If time allows, do #9174 and/or #12408 (time does not allow)

Related issues

• Related to #10034 (closed)
• Related to #6907 (closed)
• Related to #11815 (closed)
• Related to #16028 (closed)
• Related to #16123 (closed)
• Related to #16124 (closed)
• Related to #16142 (closed)
• Related to #16542 (closed)
• Related to #10442 (closed)
• Blocks #12408
• Blocks #9174
• Blocks #13284
• Blocks #13450 (closed)
• Blocks #16091
• Blocks #12113

Original created by @intrigeri on 14588 (Redmine)