Interview Questions

Shell Script - Find Unique Elements in a File

In this article, let's create a shell script to find unique elements in a file containing a list of Strings.

Consider the below file containing a list of Strings which are repeated.

Data.txt

Afghanistan
India
Albania
Algeria
Austria
Australia
Chile
Mongolia
China
France
Italy
India
Finland
Afghanistan
Dubai
Indonesia
Armenia

Look at the shell script below.

Unique.sh

#!/bin/sh
sort Sata.txt > SortData.txt
prev=0;
for f in $(cat SortData.txt | sed 's/ //g')
do
if [ $f == $prev ]; then
    echo -n ""
else
    echo "$f"
fi
prev=$f;
done

The shell script Unique.sh, initially sorts the file Data.txt in to a new file SortData.txt. Then it reads every line from the sorted file SortData.txt and finds the unique elements in the file by comparing with previous element.

Here is the output

Afghanistan
Albania
Algeria
Armenia
Australia
Austria
Chile
China
Dubai
Finland
France
India
Indonesia
Italy
Mongolia