Remove Duplicate Lines

How to use it

1. Paste your list, one item per line.
2. Turn on ‘Ignore case’ or ‘Sort result’ if you need them.
3. Copy the deduplicated list.

What is a duplicate line remover?

A duplicate line remover reads a list one line at a time and keeps each line only the first time it appears. Every later copy is dropped, so you end up with unique lines in their original order.

key = line, trimmed if Trim is on, lowercased if Ignore case is on
keep the line only if its key has not been seen before

The two options define what counts as the same line. Trim spaces, on by default, ignores spaces and tabs at the ends of a line, so a value with a trailing space matches its clean twin. Ignore case merges Apple, APPLE and apple into one entry, keeping whichever spelling came first. Differences inside a line, a period, a double space, a different dash, always keep lines separate.

Worked example

How each option removes duplicate lines

How duplicate removal works

The tool reads your text one line at a time, remembers each line the first time it appears, and drops every later copy. Order is preserved, so the first occurrence stays exactly where it was. Blank lines are removed on the way through, which means the result is a solid list of unique values with no gaps.

Two options control what counts as the same line. ‘Trim spaces’ ignores spaces and tabs at the start and end of a line, so ‘apple ’ and ‘apple’ match; it is on by default because trailing spaces are the most common reason a list still looks duplicated after cleaning. ‘Ignore case’ makes Apple, APPLE and apple one entry, and the version kept is whichever came first.

What still counts as different

Spelling and punctuation differences are never merged. An address with a trailing period is a different line from the same address without one, and ‘New York’ is different from ‘New York City’. Spaces inside a line matter too: ‘ice cream’ with one space and ‘ice cream’ with two stay separate, so run messy lists through the remove extra spaces tool first. The tool compares whole lines only; it will not find a repeated word inside a paragraph.

Compared with spreadsheets and the command line

Excel's Remove Duplicates button on the Data tab does the same job for a column, and Google Sheets has the UNIQUE function. On the command line, sort -u sorts and dedupes in one step, while uniq on its own removes only duplicates that sit next to each other. This tool keeps your original order, offers a case switch, and needs no import step, which makes it the quicker choice for a list that arrived in an email or a document. The status line reports how many unique lines remain and how many duplicates were removed, so you can sanity-check a large list before pasting it back.

Common uses

• Deduping email and mailing lists
• Cleaning keyword and tag lists
• Merging data pulled from several sources
• Preparing unique values for a spreadsheet
• Removing repeated URLs from a crawl or export
• Cleaning copied lists of names, SKUs or IDs

When to use it

• You need to remove duplicate emails from a mailing list before an export goes out twice to the same person.
• A keyword list for an ad campaign grew from several sources and half the terms repeat.
• A crawler export is full of repeated URLs and you want each page listed once.
• You want to remove duplicate lines from a pasted column without opening Excel or writing a formula.
• Two copied name lists need merging into one list with no repeats.

Related tools

Sort Lines offers more orderings than the built-in A to Z switch, including numeric and by-length sorting after you dedupe.

Remove Extra Spaces fixes double spaces inside lines, the usual reason two identical-looking entries refuse to merge.

Remove Line Breaks turns the finished unique list into a single comma-separated line for a To field or a tag box.