uniq collapse and count duplicates
uniq removes or counts lines that repeat one after another. Because it only compares neighbouring lines, it is almost always used right after sort.
Availability Preinstalled on macOS, Linux and BSD
Syntax
uniq [options] [file]Common options
| Option | What it does |
|---|---|
-c | Prefix each line with how many times it occurred |
-d | Print only lines that are repeated |
-u | Print only lines that are not repeated |
-i | Ignore case when comparing |
Examples
Remove duplicates
sort visitors.txt | uniqTop 10 most frequent lines
sort ips.txt | uniq -c | sort -rn | head -n 10Use uniq with sqx
The SQL version of sort | uniq -c
The classic counting pipeline and the GROUP BY query give the same answer, but sqx skips the header and handles quoted CSV fields.
cut -d, -f4 users.csv | sort | uniq -c | sort -rn
cat users.csv | sqx "SELECT country, COUNT(*) AS n FROM stdin GROUP BY country ORDER BY n DESC"Full documentation
This page covers the essentials. For every option, see the official documentation:
Or read the manual in your terminal: man uniq