How to grep inside while loop with two files or one file - awk

Question

I have this use case where I am trying to list some keys from s3 and filtering the results based on a grep command
fileA - abc/def
def/123
After listing the keys, I am trying to remove this exact key from the list. For example, if list return 2 other keys with the same prefix
list - abc/def/123
abc/def/1234
abc/ghi/12345
def/123/456
def/456/4567
I want to remove the keys matching the pattern read from file i.e. abc/def and def/123
Code :
while read line; do
prefix = $(echo "$line"| grep -oPw '[A-Za-z0-9]*')
aws s3api list-objects --bucket blah-bucket --prefix "$prefix" | grep -vFfw "$line" > result
done < fileA
I am getting this error that command not found : prefix
What am I missing here in the loop?

Answer 1

This is a common problem that has been addressed in various questions posted here for years and years. :-)
The notation you want should look more like this:
prefix="$(echo ...)"
Remember that the shell is a shell, not really a full fledged programming language. Its parsing rules were intended to facilitate calling other programs, and setting up the plumbing to allow those programs to interact with each other.
Here are the various ways mis-placed spaces can be interpreted by shells in the Bourne family (sh, bash, ksh, zsh, ash/dash). Consider:
var=val
var =val
var= val
var = val
var=val: this is the correct syntax for variable assigment -- an unquoted word followed immediately by an equals followed immediately by an argument.
var =val: this runs the var command with =val as its argument.
var= val: this assigns an empty string to the var variable, then runs the val command as if var had been exported to it. This is meant to provide single-use environment variables to commands called by the shell.
var = val: this runs the var command with = and val as arguments.
Other (non-Bourne-style or non-POSIX) shells will have different interpretations.
Also, beware that you will be overwriting the file result for every iteration of this loop.

Answer 2

Well, you had simple syntax error. But, if I understand the what you mean by remove the keys matching the pattern read from file, you're working too hard.
If the following solution does what you want, I guarantee it will run faster and be easier to understand:
$ head patterns input
==> patterns <==
abc/def
def/123
==> input <==
abc/def/123
abc/def/1234
abc/ghi/12345
def/123/456
def/456/4567
$ grep -vf patterns input
abc/ghi/12345
def/456/4567
Any shell solution that iterates over the data is bound to be the wrong approach. Look for ways to let grep and friends operate on whole files, and use the shell to choose the files. It always a safe bet your problem can be solved that way, because over the decades lots of problems looked like your problem. :-)

Answer 3

You can also use the following chain of commands:
$cat to_remove.in
abc/def
def/123
$cat to_process.in
abc/def/123
abc/def/1234
abc/ghi/12345
def/123/456
def/456/4567
$awk 'BEGIN{ORS="\\\\|"}{print}' to_remove.in | sed 's/\\|$//' | xargs -I {} grep -v {} to_process.in
abc/ghi/12345
def/456/4567
Explanations:
awk will be used to create a regex from file to_remove.in with | between each line that will be used by grep -v to exclude the lines from file to_process.in
sed 's/\\|$//' is used to remove the last | at the end of the regex string
then you use xargs to pass the resulting regex string to your grep command