Duplicate entries creep into any list that has been merged, exported, or copied together from several places — email lists, keyword sets, ID exports, log lines. Removing them by hand is impractical beyond a few dozen rows.
How to use the Remove Duplicate Lines
- Paste your list, one entry per line
- Choose whether matching should ignore case
- Click to remove duplicates
- Copy the deduplicated list
Case sensitivity changes the answer
With case-sensitive matching, Apple and apple are different lines and both survive. With case-insensitive matching, the second is removed and the first is kept.
Which you want depends entirely on the data. Email addresses are case-insensitive in practice, so deduplicate them ignoring case. Passwords, API keys, hashes and Base64 values are case-sensitive and must be deduplicated exactly — treating them loosely will silently destroy valid entries.
What gets kept
The first occurrence of each line is kept and later repeats are dropped, so the original ordering of the surviving lines is preserved. That matters when the list is in a meaningful order — by priority, by date, or by relevance.
Note that lines differing only by trailing whitespace are not duplicates as far as a literal comparison is concerned. If a list looks like it should deduplicate further than it does, invisible trailing spaces are almost always the reason.
Frequently asked questions
Which duplicate is kept — the first or the last?
The first. Later repeats are removed, so the relative order of the remaining lines is unchanged.
Does it sort the list as well?
No, order is preserved. If you want the result sorted, run it through the Sort Lines tool afterwards.
Can it handle very large lists?
Yes — deduplication uses a hash set, so it handles tens of thousands of lines quickly. The practical limit is the browser's memory, not the algorithm.
Why are two lines that look identical not being removed?
Almost always trailing whitespace, or a non-breaking space pasted from a web page or Word document. Both are invisible but make the lines genuinely different.