Remove Duplicate Words

The text is split wherever there is a space, tab or line break, and each piece between them is a word. A word is removed if the same piece, ignoring capitals, has already appeared anywhere earlier in the text. The first occurrence always stays.

Two details follow from that:

• Capitals don't matter. The is a repeat of the.
• Punctuation does. A comma or full stop attached to a word makes it a different piece, so red, and red are not duplicates.

In the second row, the second red, goes, but Red stays because it has no comma.

A removed word takes nothing else with it. The spaces on both sides stay, so you get double spaces wherever a word was deleted. The first row above actually comes out as the, two spaces, cat sat on, two spaces, mat, two spaces, end. If the last word of your text is removed, a trailing space is left.

Run the result through Remove Extra Spaces to close the gaps. Line breaks are also kept, so a line can end up shorter, or even empty if every word on it was a repeat.

A word only survives the first time it appears in the entire text. If you paste a list of phrases, one per line, a word repeated on a later line is removed from that line:

becomes apple banana, then cherry, then date, each still on its own line but with a leading space where the removed word was. If you want to remove whole repeated lines instead, use Remove Duplicate Lines. If you want repeats removed only within each line, run the lines through one at a time.

It is good for comma-free tag and keyword lists, hashtag lists, and text where a word got doubled by a copy-paste slip, such as the the.

It is a poor fit for normal sentences. Ordinary prose repeats words like the, and and to constantly, so after the first sentence most of them are gone and the text stops making sense. To find accidental doubled words in an essay, search for them instead of removing every repeat.