Add robots.txt to soledad
lets add a robots.txt file to soledad server with the following content:
User-agent: * Disallow: /
h2. previously
from soledad.log on clam
2014-09-15 16:53:56+0000 [-] 66.249.69.202 - - [15/Sep/2014:16:53:55 +0000] "GET /robots.txt HTTP/1.1" 401 85 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" 2014-09-15 16:53:57+0000 [-] 66.249.69.170 - - [15/Sep/2014:16:53:56 +0000] "GET /user-f74ccf0aa010f34d44c22fe034fdfd0a HTTP/1.1" 401 85 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" 2014-09-15 17:32:24+0000 [-] 66.249.69.186 - - [15/Sep/2014:17:32:24 +0000] "GET /user-c353c534c44fe26629283ff390e8d837 HTTP/1.1" 401 85 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
(from redmine: created on 2014-10-02)