How to Merge multiple files, removing duplicate values

If you have multiple files and you want to remove duplicate values from, while creating a single file.

File1 (one.txt):

123 123 234 345

File2 (two.txt):

123 678 567 890

As you can see, the two files contain some duplicates. You can easily remove these using the following command:

sort -u one.txt two.txt
Code language: Bash (bash)

This will output:

123 234 345 567 678 890

You can as easily pipe this into a third file as follows:

sort -u one.txt two.txt > three.txt
Code language: Bash (bash)

This will take the first two files (you can add as many as you like), and pipe the output into a file called three.txt.


Notify of
Inline Feedbacks
View all comments