awk to find and output differences in files - awk

Question

I am trying to find the differences between file1.txt and file2.txt and output the differences. I tried diff and sed and the output does not return any differences. I also tried awk and matching on $2, but I think the syntax is wrong as a file gets created but it is 0kb. The actual data I am using is quite large but I know there should be 18 differences. Thank you :).
awk 'NR==FNR{a[$2]++;next} !($2 in a){print $2}' file1.txt file2.txt > diff.txt
file1.txt
chr1 955542 955763
chr1 957570 957852
chr1 976034 976270
file2.txt
chr1 955542 955763 + AGRN:exon.1
chr1 957570 957852 + AGRN:exon.2
chr1 976034 976270 + AGRN:exon.2;AGRN:exon.3;AGRN:exon.4
chr1 976542 976787 + AGRN:exon.3;AGRN:exon.5
chr1 976847 977092 + AGRN:exon.6
Desired output
chr1 976542 976787 + AGRN:exon.3;AGRN:exon.5
chr1 976847 977092 + AGRN:exon.6
Diff result (since these are the two records that are not in both files)
1,52058c1,52040
< chr1 955542 955763
< chr1 957570 957852
< chr1 976034 976270

Answer 1

I'm curious while diff isn't working like you want, but your awk logic isn't correct:
You're checking the second field's (delimited by spaces) value only. In your example the second field is all identical so nothing is being printed out. Using the whole line instead works as expected:
Using your example text where all is different:
$ cat file1.txt
chr1 955542 955763
chr1 957570 957852
chr1 976034 976270
$ cat file2.txt
chr1 955542 955763 + AGRN:exon.1
chr1 957570 957852 + AGRN:exon.2
chr1 976034 976270 + AGRN:exon.2;AGRN:exon.3;AGRN:exon.4
$ awk 'NR==FNR{a[$0]++;next} !($0 in a){print $0}' file1.txt file2.txt > diff.txt
$ cat diff.txt
chr1 955542 955763 + AGRN:exon.1
chr1 957570 957852 + AGRN:exon.2
chr1 976034 976270 + AGRN:exon.2;AGRN:exon.3;AGRN:exon.4
Here's with the second line identical just to show it working in a more obvious way.
$ cat file1.txt
chr1 955542 955763
chr1 957570 957852
chr1 976034 976270
$ cat file2.txt
chr1 955542 955763 + AGRN:exon.1
chr1 957570 957852
chr1 976034 976270 + AGRN:exon.2;AGRN:exon.3;AGRN:exon.4
$ awk 'NR==FNR{a[$0]++;next} !($0 in a){print $0}' file1.txt file2.txt > diff.txt
$ cat diff.txt
chr1 955542 955763 + AGRN:exon.1
chr1 976034 976270 + AGRN:exon.2;AGRN:exon.3;AGRN:exon.4
EDIT
Based on a comment stating:
"There should be 18 differences out of 52,000 lines. File1.txt is 52,058 entries and file2.txt has 52,040 entries in it. I am trying to find out what the 18 are"
Given you said file1 has more lines, you need to process file2 first. The first file read is populating the array and then the second is checking for lines existing in that array. You need to process the smaller file first so that the additional lines you're interested in aren't in the array. It'd be the same logic above, just with the file order switched, e.g.:
$ cat file1.txt
chr1 955542 955763
chr1 957570 957852
chr1 976034 976270
New Line!
Not in file2!
$ cat file2.txt
chr1 955542 955763 + AGRN:exon.1
chr1 957570 957852
chr1 976034 976270 + AGRN:exon.2;AGRN:exon.3;AGRN:exon.4
$ awk 'NR==FNR{a[$0]++;next} !($0 in a){print $0}' file2.txt file1.txt > diff.txt
$ cat diff.txt
chr1 955542 955763
chr1 976034 976270
New Line!
Not in file2!
$ awk 'NR==FNR{a[$0]++;next} !($0 in a){print $0}' file1.txt file2.txt > diff.txt
$ cat diff.txt
chr1 955542 955763 + AGRN:exon.1
chr1 976034 976270 + AGRN:exon.2;AGRN:exon.3;AGRN:exon.4
Note that reading file1 first doesn't emit the additional lines.
If you don't care about the additional text on the lines, just the text in the second field, then you could use $2 as you originally did.

Answer 2

$ awk 'NR==FNR{a[$2];next} !($2 in a)' file1 file2
chr1 976542 976787 + AGRN:exon.3;AGRN:exon.5
chr1 976847 977092 + AGRN:exon.6