Common uses
- Mailing and contact lists: merge exports and remove repeated email addresses or phone numbers before a send
- Records: order numbers, ticket IDs, customer IDs, UUIDs and Discord IDs
- IT: IP blocklists, user agents, hostnames and log lines (see also Sort IP addresses)
- SEO: keyword lists, tags and URL lists
- Inventory: product names and SKUs merged from several sources
Want to know how many times each item appeared? Use Count duplicates.
Options
- Ignore capitalization treats “Paris” and “paris” as the same item. The first spelling is kept.
- Ignore spaces at the ends treats “apple” and “apple ” as the same item. It’s on by default, because stray spaces are invisible and almost never intentional.
Blank lines are removed automatically. The order of your list is kept. To sort as well, press A → Z afterwards.
Find what’s shared between two lists
If you have two lists and want the items they have in common, or the ones that only appear in one, use Compare two lists.
Keep the first of each
apple banana apple cherry banana apple
apple banana cherry
Ignoring capitalization
Paris LONDON paris London Rome
Paris LONDON Rome
Questions
Which copy is kept?
The first one. Later repeats are deleted.
How do I count how many duplicates there were?
The message under the box tells you how many duplicates were removed and how many items are left.
Can I remove duplicates from a comma separated list?
Yes. Commas are detected automatically when the list is on one line, or you can choose the separator yourself.
Does it work on big lists?
Yes. A list of 100,000 lines is handled in a fraction of a second, all in your browser.