known_hosts truncation when filesystem full
i ran out of space on the filesystem where i was using @monkeysphere ssh-proxycommand@, and @~/.ssh/known_hosts@ ended up getting truncated somehow, along with a lot of spew of grep complaining (sadly, i don't have access to the spew any more).
Because the truncation was in the middle of a key, it meant that future runs of @ssh-keygen -F $hostname@ would fail, which in turn would cause @monkeysphere ssh-proxycommand@ to fail, due to it being @set -e@. this meant that every attempt to ssh would end in:
ssh_exchange_identification: Connection closed by remote host
looking at the truncated file, i see that it was truncated at position 0x28000 -- very likely a filesystem block boundary, since 4KiB blocks are size 0x1000. Then i see another key slapped right in the middle of the line.
advice to try to replicate:
- i have a large known_hosts file -- on the order of 270KiB -- i suspect that contributed to it being likely to screw up somewhere
- i'd start by trying to replicate the truncation with just @monkeysphere update-known_hosts@ on a mostly-full filesystem.
workaround:
i resolved the problem for myself by cleaning up the @known_hosts@ file (i restored from a backup). you could probably also recover by cleaning out the truncated line (though you'd lose not only that key, all of the keys that come after it)
observations:
it seems likely that some thing is appending to @~/.ssh/known_hosts@ -- if there is no trailing newline in the file, appending is not a good idea, since it would just append the data to the tail of such line. for an otherwise well-formed line with a comment that is simply missing a newline, this would look like an extra-long comment (and the new line would be ignored). for a line that is already truncated in the middle of the key, or has no comment at all, it would make the key unintelligible.
I backed up the corrupted file for future examination. here's an example of the failing output.
0 dkg@pip:~$ ssh-keygen -f ~/.ssh/known_hosts.busted -F no.such.host key_read: uudecode AAAAB3NzaC1yc2EAAAABIwAAAQEAtoXLV1PpBM+Ad+WKUri/0kJyQwKUSJkcum3WiRHpV4j69BG5bWC0D+HKuPfYP5BXBOFaFe2lxNVyOP03bOFvoA4UHUL3l/iBXNqnn8AmkLFHUVJv3/CS8K+XirIqoPUddkaOkRYK1WAEVmIg6GH7z1V3xabfTGmik4LE6kPD+YZ4kuYezpYgLhYYX8x0Rimp0ac+yvIwL5LNOl8uo5DK+VNWFIOD2PRgmocJn/YXH|1|oS9U8neREGPKvjnXgFINWjM/uOg=|Q0PPWPo9fg/8FpSVG5YET3QPUk4= ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDasn6qazShJv/hnFpwmo+JJZrm8ZJAvOCYQfZ9uL8p1EDTSgWvla0g3ggJJNS1UEAgoLA6Oly/TR5llRxcEavXH1YU4gMLo9X8BDBNiTwZZ5AlBS3PGNk5jfWGsSBiibMTLJ/uhVOdDm/rjI6Vt1ifGEBcVwFhVfIw87zGK0MPjAwKnluaH8NbpxKkS++6DHcT4cr6QKVkGfkmiqMAyASzo7AzMLyDBBtbnPh3/RgBr/yWFufGhI+L4rNHpoDGnfjl0lQWtS2RSqkh9kdXQr/IF+xkxLMxFcBpJArV8zdlgLzlMoN1v1iF/QjWZaIiS0V/xum+4to/UnH+y50JxKvN failed line 544 invalid key: some.host.example.org... /home/dkg/.ssh/known_hosts.busted is not a valid known_hosts file. 1 dkg@pip:~$
(from redmine: created on 2010-02-09)