Delete duplicated lines based in a pattern - awk

Question

I'm trying to remove all duplicated lines based in a specific pattern, below you can see a small sample of my file:
Jul 22 12:43:57 localhost haproxy[24190]: <IP_ADDRESS>:56232 [22/Jul/2021:12:43:57.862] http-ap~ rpc-ap/localhost 0/0/0/3/3 200 402 - - ---- 110/108/8/8/0 0/0 "POST / HTTP/1.1" getAccountInfo api.mainnet-beta.solana.com foundation chrome-extension://adlbhecpgmnfllbdnbhlk chrome-extension://adlbhecpgmnfllbdnbhlkobnbpdkcdod application/json
Jul 22 12:44:02 localhost haproxy[24190]: <IP_ADDRESS>:56232 [22/Jul/2021:12:44:02.436] http-ap~ rpc-ap/localhost 0/0/0/0/0 200 400 - - ---- 111/109/4/4/0 0/0 "POST / HTTP/1.1" getTokenAccountsByOwner api.mainnet-beta.solana.com foundation chrome-extension://adlbhecpgmnfllbdnbhlk chrome-extension://adlbhecpgmnfllbdnbhlkobnbpdkcdod application/json
Jul 22 12:44:03 localhost haproxy[24190]: <IP_ADDRESS>:56232 [22/Jul/2021:12:44:03.033] http-ap~ rpc-ap/localhost 0/0/0/0/0 200 402 - - ---- 111/109/4/4/0 0/0 "POST / HTTP/1.1" getAccountInfo api.mainnet-beta.solana.com foundation chrome-extension://adlbhecpgmnfllbdnbhlk chrome-extension://adlbhecpgmnfllbdnbhlkobnbpdkcdod application/json
Jul 22 14:11:56 localhost haproxy[24190]: <IP_ADDRESS>:37086 [22/Jul/2021:14:11:56.959] http-ap~ rpc-ap/localhost 0/0/0/0/0 200 400 - - ---- 144/142/4/4/0 0/0 "POST / HTTP/1.1" getTokenAccountsByOwner api.mainnet-beta.solana.com foundation chrome-extension://bfnaelmomeimhlpmgjnjo chrome-extension://bfnaelmomeimhlpmgjnjophhpkkoljpa application/json
Jul 22 14:11:57 localhost haproxy[24190]: <IP_ADDRESS>:37086 [22/Jul/2021:14:11:57.559] http-ap~ rpc-ap/localhost 0/0/0/3/3 200 402 - - ---- 136/134/4/4/0 0/0 "POST / HTTP/1.1" getAccountInfo api.mainnet-beta.solana.com foundation chrome-extension://bfnaelmomeimhlpmgjnjo chrome-extension://bfnaelmomeimhlpmgjnjophhpkkoljpa application/json
The only information I care is the chrome-extension ID for example: chrome-extension://adlbhecpgmnfllbdnbhlkobnbpdkcdod
Note:if you guys notice I have 2 patters with chrome-extension://something in the same line, one with ID truncated and another with the ID complete, I wish to perform this operation matching the complete one.
in this example above I have only 5 entries, but in my real file a have a lot of them, I would like to find "chrome-extension://" and remove all lines that have the value of it duplicated.
In this example above my result could be:
Jul 22 12:44:03 localhost haproxy[24190]: <IP_ADDRESS>:56232 [22/Jul/2021:12:44:03.033] http-ap~ rpc-ap/localhost 0/0/0/0/0 200 402 - - ---- 111/109/4/4/0 0/0 "POST / HTTP/1.1" getAccountInfo api.mainnet-beta.solana.com foundation chrome-extension://adlbhecpgmnfllbdnbhlk chrome-extension://adlbhecpgmnfllbdnbhlkobnbpdkcdod application/json
Jul 22 14:11:57 localhost haproxy[24190]: <IP_ADDRESS>:37086 [22/Jul/2021:14:11:57.559] http-ap~ rpc-ap/localhost 0/0/0/3/3 200 402 - - ---- 136/134/4/4/0 0/0 "POST / HTTP/1.1" getAccountInfo api.mainnet-beta.solana.com foundation chrome-extension://bfnaelmomeimhlpmgjnjo chrome-extension://bfnaelmomeimhlpmgjnjophhpkkoljpa application/json
But it could be any of the 5 entries above, I don't care about which line it will be, it could be sorted, the important thing is to get only one line for each chrome ID.
I tried a few things with awk, sed, sort and uniq, but I'm a noob in Linux and regex, so I would be really grateful if you guys could give me a light.

Answer 1

If the input is sorted, uniq can be told to ignore the first 24 fields using the -f option.
sort file-sample | uniq -f 24

Answer 2

This might work for you (GNU sed):
sed -E '/chrome-extension/{G;/chrome-extension(\S* ).*\1/d;P;s/chrome-extension//g;h;d}' file
If a line contains chrome-extension, append previous extensions and try and match.
If a match is found, a duplicate exists, so delete the line.
Otherwise, print the line, remove the literal(s) chrome-extension, make a copy and then delete the line.

Answer 3

$ awk '!seen[$25]++' file
Jul 22 12:43:57 localhost haproxy[24190]: <IP_ADDRESS>:56232 [22/Jul/2021:12:43:57.862] http-ap~ rpc-ap/localhost 0/0/0/3/3 200 402 - - ---- 110/108/8/8/0 0/0 "POST / HTTP/1.1" getAccountInfo api.mainnet-beta.solana.com foundation chrome-extension://adlbhecpgmnfllbdnbhlk chrome-extension://adlbhecpgmnfllbdnbhlkobnbpdkcdod application/json
Jul 22 14:11:56 localhost haproxy[24190]: <IP_ADDRESS>:37086 [22/Jul/2021:14:11:56.959] http-ap~ rpc-ap/localhost 0/0/0/0/0 200 400 - - ---- 144/142/4/4/0 0/0 "POST / HTTP/1.1" getTokenAccountsByOwner api.mainnet-beta.solana.com foundation chrome-extension://bfnaelmomeimhlpmgjnjo chrome-extension://bfnaelmomeimhlpmgjnjophhpkkoljpa application/json
or if you prefer:
$ sort -k25,25 file | uniq -df24
Jul 22 12:43:57 localhost haproxy[24190]: <IP_ADDRESS>:56232 [22/Jul/2021:12:43:57.862] http-ap~ rpc-ap/localhost 0/0/0/3/3 200 402 - - ---- 110/108/8/8/0 0/0 "POST / HTTP/1.1" getAccountInfo api.mainnet-beta.solana.com foundation chrome-extension://adlbhecpgmnfllbdnbhlk chrome-extension://adlbhecpgmnfllbdnbhlkobnbpdkcdod application/json
Jul 22 14:11:56 localhost haproxy[24190]: <IP_ADDRESS>:37086 [22/Jul/2021:14:11:56.959] http-ap~ rpc-ap/localhost 0/0/0/0/0 200 400 - - ---- 144/142/4/4/0 0/0 "POST / HTTP/1.1" getTokenAccountsByOwner api.mainnet-beta.solana.com foundation chrome-extension://bfnaelmomeimhlpmgjnjo chrome-extension://bfnaelmomeimhlpmgjnjophhpkkoljpa application/json