Remove Duplicate Lines
Keeps the first occurrence of each line, reports how many were removed.
—
Worked example
With case-insensitive matching, apple and Apple count as the same entry. Turning it off keeps both — correct when the capitalisation is meaningful, wrong when it is a paste artefact.
Duplicate lines come from three places: concatenated exports, copy-paste from a paginated view, and case or whitespace differences that look identical on screen but differ byte for byte. This tool handles all three, and the trade-off is always which of the duplicates you keep.
Keep first, or keep last?
This keeps the first occurrence. That is the right default for append-only exports where later rows are the stale ones, and the wrong default for update logs where the last row is current. If you need the latest value, reverse the list, dedupe, and reverse back.
Whitespace
Trimming before comparison catches the most common invisible difference — a trailing space from a spreadsheet column. The output is the trimmed line, so the result is also cleaner than the input.
What counts as a duplicate
Exact matches after the chosen normalisation. It does not catch near-duplicates such as "New York" versus "New York, NY", or URLs that differ only by a tracking parameter. Those need a real matching rule, not a dedupe button.