mayurilahane authored 5 years ago and ulif committed 5 years ago

Many thanks to @mayurilahane for spotting this typo. Also small contributions are needed to make better software :)

drebs authored 5 years ago and ulif committed 5 years ago

Wordlist in pt-br was first introduced in 7743ed55. The differences to
this one are:

  - 9-characters words are introduced.
  - suffixes removal is made after accounting for popularity.
  - less frequent words that differ only in the last character are
    removed.

The current pt-br wordlist was generated as follows:

  1. Download a dump of portuguese Wikipedia pages, process all pages
     and determine the frequency of each word.
  2. Start from /usr/share/dict/brazilian and filter out:
       - words not matching /^[a-z]+$/,
       - words shorter than 4 characters, and
       - words longer than 9 characters.
  3. Sort remaining words using pt Wikipedia frequencies.
  4. Take the top 30K words (just because after filtering we still get
     roughly the amount we need).
  5. Filter out:
       - all words that are a suffix of any other word in the list.
       - less frequent words that differ only by the last character.
  6. Take the 7776 most frequent words.

No further curation was made.

drebs authored 5 years ago and ulif committed 5 years ago

The wordlist was generated from 2 different sources of words:

  - The file /usr/share/dict/brazilian from Debian's wbrazilian package.
  - A dump of the pages of Wikipedia in portuguese.

The final pt-br wordlist was generated as follows:

  1. Download a dump of portuguese Wikipedia pages, process all pages
     and determine the frequency of each word.
  2. Start from /usr/share/dict/brazilian and filter out:
       - words not matching /^[a-z]+$/,
       - words shorter than 4 characters, and
       - words longer than 8 characters.
  3. Remove all words that are a suffix of any other word in the list.
  4. Sort remaining words using pt Wikipedia frequencies.
  5. Take the 7776 most frequent words.

No further curation was made.

There are obvious drawbacks in this approach (eg: many very frequent
words are left out because they are either too short or too long or
contain accents or cedilla), but it was the best cost-benefit i could
think about.

`pycodestyle` complained about non raw-string expressions.

We received an official permission from Arnold G Reinhold to use the
name 'diceware' under the following conditions:

1) we use Pythons `system.random` as default source of randomness
2) we continue to provide support for real dice
3) we do not use the name for other tools than this commandline tool
4) we include the trademark statement commited hereby.

Many thanks go to Mr. Reinhold.

Fixes #48.

When calling `logger.removeHandler(h)` with one of our custom
`NullHandler` instances, things crashed.

Using `IOError` for compatibility, from Python 3.3 `IOError` is alias of `OSError`
https://stackoverflow.com/a/28633573/6202405

Checking the errno of 'No such file or directory' so that only that exception will be catched.
https://docs.python.org/2/library/errno.html
https://stackoverflow.com/a/39414133



Signed-off-by: Bhavin Gandhi <bhavin7392@gmail.com>

Normal errors should be displayed by default, even if they are not
catastrophic.

The documenting comment (starting with ``#: ``) is neccessary for vars
and constants only. All these comments are used by `Sphinx` to generate
documentation.

Put consts at top of file and functions below. The code is better
readable this way.

The path to wordlist files is hardcoded in the generated sphinx
documentation and because it may differ depending on how the package is
installed this creates different problems:

  - the path hardcoded in the online api documentation may differ from
    the one in each installation, leading to incorrect information
    provided to the user of the api.

  - packaging diceware documentation to specific operating systems may
    generate unreproducible packages.

This commit replaces the variable WORDLISTS_DIR by a function called
get_wordlists_dir() that delays the construction of the path instead of
doing it during module loading.

We do not accept open filedescriptors as argument for `WordList`
anymore. This seems to be the only way to make sure we can open/close
wordlist files whenever we want.

The `argparse.FileType` type is considered bad style. Cf.
http://bugs.python.org/issue13824. With this FileType we cannot tell
when to close a file after reading (which should be done for most
regular wordlists) or not (in case of input from stdin).

`WordList` accepts both, already opened files and also file paths. As
this class can better care for closing open files after use, we should
not open files if not neccessary.

Files already opened by third-parties should be closed by them.

Since Python3.2 SafeConfigParser is an alias for ConfigParser and
emits warnings when imported. We want to avoid these warnings.

Logging was available all the time but not yet activated. We now
enable logging when the main script is executed. This makes the
`verbose` commandline option functional.

When using `realdice` randomness source and picking values out of short
sequences (sequences shorter than the number of dice sides), we might
require additional dice throws (if a value rolled is not in range of
sequence length). We now print a hint when this happens.

We avoid the modulo operation now, if it comes to small sequences.
Instead we discard invalud rolls and require new ones.

Sequences with lenths != s**n (with s:= number of dice sides, n := any
natural number) require an extra roll.

The number of rolls is cruical for the whole `choice` distribution and
a specialized function is much better testable.

The EFF list is currently certainly the scientifically most advanced
list. Thanks to @anarcat for proposing this switch! The securedrop
list is still available.

RealDiceRandomSource and its tests updated to expect argparse.Namespace objects instead of dicts (fix for using the 'realdice' randomness source)

Make the tests succeed. In wordlists we now also accept

  '1-2-3-4 someterm'

for numbered entries. We still support '1234 somterm'.

The Electronic Frontier Foundation published own wordlists, which
were, as it looks, carefully assembled. See

https://www.eff.org/deeplinks/2016/07/new-wordlists-random-passphrases

for details.

We ship now with the long list.