Remove Duplicate Lines
Remove duplicate lines, keeping the first of each in order.
What you provide
How this calculation works
The tool keeps the first time each line appears and drops later repeats, leaving the remaining lines in their original order. Case-insensitive mode treats lines that differ only in capitalisation as duplicates.
What the results mean
- Order preserved
- Unlike a sort-and-unique, this keeps your lines in the order they first appeared.
- Ignore case
- Treats "Apple" and "apple" as the same line, keeping whichever came first.
- Duplicates removed
- How many lines were dropped. Compare it against the original line count to sanity-check the result before using it.
- Trailing whitespace
- Two lines that differ only by a trailing space are not duplicates to a computer. If fewer lines were removed than you expected, that is usually why.
Common problems and fixes
- Lines that look identical were not removed
- Something invisible differs — a trailing space, a tab where the other line has spaces, or a Windows carriage return on one line and not the other after mixing files from different systems. Run the text through the whitespace cleaner first, then deduplicate; the two steps together catch nearly all of these.
- I need the duplicates themselves, not the unique lines
- This tool returns the deduplicated list rather than the repeats. To find which entries were duplicated, compare the before and after counts for the scale of the problem, and sort the original first so identical lines sit next to each other and can be read off directly.
- My CSV lost rows I needed
- Deduplication works on whole lines, so two genuinely different records that happen to be identical as text — the same order placed twice on different dates, if the date is not in the row — collapse into one. For structured data, deduplicate on a key column in a spreadsheet rather than on the whole line.
Frequently asked questions
Does this sort my lines?
No, deliberately. It removes duplicates while keeping the original order, because sorting would destroy information you may care about. If you want sorted output, sort first and then deduplicate.
Which copy of a duplicate is kept?
The first one, and everything else stays in its original position. That matters for anything where the order carries meaning — a log, a ranked list, a CSV with a header row — because a sort-and-unique approach would silently reorder all of it while appearing to do the same job.
Does it handle very large lists?
Yes. The comparison uses a hash set rather than checking every line against every other, so the work grows in proportion to the number of lines rather than to the square of it. Lists in the hundreds of thousands of lines are processed in about a second, bounded by your machine's memory rather than by any upload limit.
Put this on your own site
Free to embed, no attribution required beyond the source link the frame carries itself. It runs entirely in your visitor's browser, sets no cookies and loads no third-party script.
<iframe src="https://runthetests.com/embed/remove-duplicate-lines/" width="100%" height="560" style="border:1px solid #e5e5e5;border-radius:8px" title="Remove Duplicate Lines" loading="lazy"></iframe>
Preview it at https://runthetests.com/embed/remove-duplicate-lines/. Embedded pages are marked noindex, so yours stays the canonical copy — not this one.