Normalize URLs

[osteele]

For example, entries for HTTP://www.Example.com/ and http://www.example.com/ , http://www.example.com/%7Eusername/ and http://www.example.com/~username/, http://www.example.com:80/bar.html and http://www.example.com/bar.html, HTTP://xn--xample-hva.com:80/ and http://êxample.com/ , should be de-duped.

See https://en.wikipedia.org/wiki/URL_normalization

Should "Normalizations that usually preserve semantics", e.g. http://www.example.com/alice and http://www.example.com/alice/, be de-duped? You could HEAD them and see if one redirects (at least permanently; also temporarily?) to the other.