To remove duplicate lines from a file use the command:
uniq in_file out_file
uniq searches the in_file for consecutive duplicate lines -a line that is immediately followed by one or more lines that are identical to it- and removes them. The result is an out_file that contains no duplicate lines.
Use the sort command if you want to remove duplicate lines that occur in different parts of a file.