Shell Script - Find Unique Elements in a File

By arunraj

In this article, let's create a shell script to find unique elements in a file containing a list of Strings.

Consider the below file containing a list of Strings which are repeated.

Data.txt

Afghanistan
India
Albania
Algeria
Austria
Australia
Chile
Mongolia
China
France
Italy
India
Finland
Afghanistan
Dubai
Indonesia
Armenia

Look at the shell script below.

Unique.sh

#!/bin/sh
sort Sata.txt > SortData.txt
prev=0;
for f in $(cat SortData.txt | sed 's/ //g')
do
if [ $f == $prev ]; then
echo -n ""
else
echo "$f"
fi
prev=$f;
done

The shell script Unique.sh, initially sorts the file Data.txt in to a new file SortData.txt. Then it reads every line from the sorted file SortData.txt and finds the unique elements in the file by comparing with previous element.

Here is the output

Afghanistan
Albania
Algeria
Armenia
Australia
Austria
Chile
China
Dubai
Finland
France
India
Indonesia
Italy
Mongolia

Technology:

Shell script

Search

Disclaimer Notice

THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS OF TECHDIVE.IN ''AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS OF TECHDIVE.IN BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. ANY COPYRIGHT INFRINGEMENT ON THE ARTICLES/CONTENT POSTED IN TECHDIVE.IN IS PURELY UN-INTENTIONAL AND NO LIABILITY CAN BE MADE AGAINST TECHDIVE.IN OR ITS CONTRIBUTORS.

The trademarks like Java, Spring, Spring Framework, Spring Web Flow, Oracle, SNMP, Shell Scripting, Log4j and other trademarks are property of their respective owners.

Please read our terms of use before using the site.