Cdtjkyj

I've got 2 files, both with numerous lines that only contain one number. I'm trying to see if any number from file1 matches a number in file2. This is what I tried, and for some reason it doesn't work:

for i in $(cat file1); do grep ${i} file2; done

Fore reference here is data from file1 and file2

file1   file2

2134    1251

2135    5626

5342    4327

6456    8453

3413    4537

4525    3533

2347    5738

1235    1235

7453    3462

So shouldn't this command take each line from file 1 and grep it against the whole of file2? In that case, shouldn't a match be printed on screen?

Given two ordinary Unix text files, your shell loop prints

1235

since this is the line that occurs in both files. If it does not, then one of your files may be a DOS text file. You can convert DOS text files into Unix text files with the dos2unix utility.

There is nothing major wrong with your loop given the type of data that you have, apart from the fact that it calls grep once for every line in file1. It also would match substrings, for example 100 in 1001, and it would, if any line in file1 contained spaces or tabs, split these lines into multiple words (due to the for i in $(cat ...) where the $(cat ...) is unquoted).

If you want to solve your issue this way (with a loop), you would better do

while IFS= read -r word; do

    grep -xF -e "$word" file2

done <file1

The -x and -F are explained later in my answer, and -e signifies that the next argument is the pattern to match with (otherwise, it may be taken as a command line option if it starts with a dash (-).

This would still execute grep once for each line in file1, but it would do it correctly.

To extract lines in file2 that exactly correspond to line in file1, without using a shell loop, you would use

$ grep -xF -f file1 file2

1235

This is assuming that file1 contains a reasonable number of lines, but not too many ("too many" will depend on the amount of memory that you have).

The command uses grep with -x, which forces matches across full lines only (no substring matches), and with -F which changes grep to do string comparisons rather than regular expression matches.

The -f file1 instructs grep to read the patterns (the strings to match with) from file1.

For really massive amounts of data, it would be hugely inefficient to use grep though. Instead, for this task and with this type of data (single words on individual lines), it would be better to do a relational join operation between the files:

$ join file1 file2

1235

This would, assuming that both files are lexicographically sorted, return the numbers that are the same between both files.

Using comm:

$ comm -1 -2 file1 file2

1235

comm also compares sorted files and can easily handle very large datasets. It prints three columns by default:

1. lines that occur in the first file only


2. lines that occur in the second file only


3. lines that occurs in both files

With -1 we turn off the output of the first column, and with -2 we disable the second column, leaving comm to only output the lines that are the same in both files.

You simply need to use grep -f file1 file2 OR you may also use cat file1 | grep -f /dev/stdin file2