How to use the deduplicate and sort tool
Deduplicating and sorting a list is a small task that shows up constantly: cleaning up an email list, merging several keyword exports, comparing product SKUs across two systems. Doing it by eye in a spreadsheet is slow, and Excel's "remove duplicates" is a blunt instrument that ignores case and whitespace nuances. This tool works on text you paste in, one entry per line, with every rule set by you.
What the controls do
- Deduplicate: keep only the first occurrence of each line and drop the rest. Turn it off when you only want to sort.
- Sort order: leave the order as-is, sort A to Z, or sort Z to A. Sorting uses locale-aware comparison, so accented Latin letters and numbers sort the way a human would expect rather than by raw character code.
- Case handling: choose whether
Appleandapplecount as the same entry. Treating them as the same is what you usually want for email addresses and tags; keeping them distinct matters for identifiers and code. - Trim whitespace: strip spaces and tabs from both ends of each line before comparing. This catches the invisible difference that makes two lines look identical but compare as different.
- Drop empty lines: remove blank lines from the output, so a stray carriage return doesn't survive into your cleaned list.
- Keep only duplicates: invert the usual behaviour — output only the values that appeared more than once. This is how you find what's actually repeated in a big list, which is often the real question.
- Show counts: append how many times each value occurred, useful when you're auditing a list rather than cleaning it.
- Split on: besides line breaks, you can split on commas, semicolons, spaces or tabs, so a single CSV-style line turns into one entry per cell.
Typical workflows
Merging two keyword lists. Paste list A, then list B on the lines below, tick deduplicate and sort A to Z, and you have the union.
Finding what two lists share. Paste both, tick "keep only duplicates" and "show counts". Anything appearing twice is common to both — as long as the two lists don't themselves contain internal repeats.
Cleaning an exported contact list. Tick trim whitespace, case-insensitive matching and drop empty lines. That combination removes the three most common sources of near-duplicates in a spreadsheet export.
A note on what "duplicate" means here
Comparison is exact after the options you chose are applied. So New York and New York (with a trailing space) are duplicates once trimming is on, but New York and new york only merge if you're ignoring case. If two lines look the same and aren't being merged, the difference is almost always invisible: a trailing space, a non-breaking space, or a full-width versus half-width character.
Frequently asked questions
Does it keep the first occurrence or the last?
The first. When deduplicating without sorting, the order of the output follows the order of the input and the earliest occurrence survives. That means if you paste new data after old data and something was updated, the old value wins — put the newer list first when that matters.
How is the sort ordered?
Using locale-aware string comparison. Digits and symbols generally come before letters, and letters are ordered alphabetically ignoring case distinctions rather than by ASCII code. That's what makes the result look "right" to a person, at the cost of not matching a strict byte-order sort.
Can it handle a large list?
Yes — tens of thousands of lines is fine, since the work is a linear pass plus a sort. The practical limit is the browser's ability to render the output area, not the algorithm.
Is my data uploaded?
No. Everything runs in your browser and the text you paste never leaves your device. You can confirm this by watching the Network panel in F12 while you use it.
What's the difference between "deduplicate" and "keep only duplicates"?
They're opposites. Deduplicate gives you each unique value once — a cleaned list. Keep-only-duplicates gives you the values that appeared more than once — a list of problems. Run one, then the other, depending on whether you're tidying data or auditing it.