Remove Duplicate Lines
Delete duplicate lines from a list while keeping the original order — great for cleaning emails, URLs and keywords.
What counts as a duplicate line
Two lines are duplicates only when they match character for character. The tool keeps the first copy of each line in its original position and drops every later copy, so the order of your list never changes.
That exact-match rule has two consequences people trip over:
- Case matters.
appleandAppleare different lines, so both stay. - Invisible spaces matter. A line that ends in a space, or starts with a tab, is different from the same line without it.
Here is a real run. The fourth line is apple followed by a space:
| Input | Output |
|---|---|
| apple banana Apple apple (with a trailing space) banana cherry apple | apple banana Apple apple (with a trailing space) cherry |
Seven lines become five. The second banana and the last apple are removed, but Apple and the version with the trailing space survive because they are not identical.
Blank lines are treated as duplicates too
An empty line is a line like any other. The first blank line is kept, and every blank line after it is removed. If your text is split into paragraphs, the paragraphs after the first gap run together:
becomes
So use this tool on lists, not on prose. If you want to delete all blank lines instead of all but one, use Remove Empty Lines.
Making near-duplicates match before you run it
If your list came from several sources, clean it first so that lines which mean the same thing are also written the same way:
- Run it through Remove Extra Spaces. That strips spaces and tabs from the start and end of every line and collapses runs of spaces inside a line.
- If case shouldn't matter, as with email addresses, keywords or domain names, convert everything to lowercase first.
- Paste the cleaned text here.
Windows line endings are not a problem. When you paste text into the box, the browser stores every line break as a single line feed, so a line copied from a Notepad file matches the same line copied from a Mac or Linux file.
Lists it works well on
- Email lists merged from two exports, after lowercasing.
- Keyword lists from several research tools, where the same phrase appears many times.
- URL lists from a crawl or a log file. Note that
/pageand/page/are different lines, so normalise trailing slashes first if they point to the same page. - Log lines, but only if they don't start with a timestamp, since a different time makes every line unique.
Frequently asked questions
Will it sort my list as well?+
No. The remaining lines stay in the order they first appeared. If you want them alphabetical, run the result through Sort Lines Alphabetically afterwards. Sorting first and removing duplicates second gives the same set of lines, just in a different order.
Can I paste a column from Excel or Google Sheets?+
Yes. A copied column pastes as one cell per line. If you copy several columns, each row pastes as one line with tabs between the cells, and a row only counts as a duplicate when every cell matches.
Does it keep the first copy or the last one?+
The first. Each line stays where it first appears and later repeats are dropped, so if your list is ranked or dated, the earliest or highest entry is the one you keep.
Is my list uploaded anywhere?+
No. The duplicates are removed by code running in your browser tab, so the text is not sent to a server. Very long lists are limited only by your device's memory.
Embed this tool on your website
Free to use on blogs, school sites and intranets. Paste this code into your page's HTML; please keep the credit line.
<iframe src="https://utilityapp.lat/embed/text/remove-duplicate-lines" title="Remove Duplicate Lines" width="100%" height="600" style="border:0;max-width:760px" loading="lazy"></iframe> <p style="font-size:13px"><a href="https://utilityapp.lat/text/remove-duplicate-lines">Remove Duplicate Lines</a> by UtilityApp</p>