How to get id's from a list that are NOT in a table? - sql

Question

I am receiving a list of ID's. Most of these already exist in a table. I need to find which ID's are NOT in the table. This question has nothing to do with joins.
My API will receive a list of IDs, such as: [1, 2, 3, 4, 5]
Let's say there are three records in the table: [2, 3, 4]
The result I'm looking for is the array: [1, 5]
Our SQL brains jump quickly to something like the following, but clearly that's not what we need:
select * from widgets where id not in [list]
We don't need the records not in the list, we need the part of the list not in the records!
My fallback is to retrieve all records in the list and subtract from the list, something like this:
existing_ids = Widget.where(id: id_list).pluck(:id)
new_ids = id_list - existing_ids
That will work...but feels heavy-handed. Particularly if id_list has 100,000 records, and the table has 99,999 of those records.
I've searched around, and the only similar result is ID from list that is not in a table ... which did not find a viable solution.
Is there any way to do this in a single SQL query? (Bonus points for an ActiveRecord solution!)

Answer 1

To compare the lists to each other, either the input list needs to go into the database or the list of existing ids needs to come out of the database. The latter you already tried and didn't like, so here's an alternative
SELECT "id" FROM unnest('{1,2,3,4,5}'::integer[]) AS "id" WHERE "id" NOT IN (SELECT "id" FROM "widgets");
Not sure about performance.

Answer 2

Depending how many records are in your database, the simplest thing might just be to select all of the IDs and then drop the duplicates in Ruby.
from_api = [1,2,3,4,5]
existing = Widgets.pluck(:id) # => [2,3,4]
from_api.difference(existing) # => [1,5]
Obviously, if you have a substantial dataset, this will be less than optimal.

Answer 3

This should work.
from_api = [1,2,3,4,5]
existing = Widgets.order(:id).ids # => [2,3,4]
new_ids = []
from_api.each{ |n| new_ids << n unless existing.include? n }
new_ids # => [1,5]
or
from_api = [1,2,3,4,5]
existing = Widgets.order(:id).ids # => [2,3,4]
from_api.map{ |n| n == existing.first ? (nil if existing = existing.drop(1)) : n }.compac # => [1,5]

Answer 4

Balancing the complexity (to the current and future developers) of unset approach, I decided for my project that the simpler approach was warranted. While I didn't profile performance, I believe any gains would be minimal, if any.
Here is the solution I ended up with:
class Widget < ApplicationRecord
def self.absent(names)
uniq_names = names.uniq
uniq_names - where(name: uniq_names).pluck(:name)
end
end
And tests:
describe '.absent' do
subject { described_class.absent(names) }
let!(:widget1) { create(:widget, name: 'old-1') }
let!(:widget2) { create(:widget, name: 'old-2') }
let(:names) { %w[new-2 old-2 new-1 old-1 new-1 old-1] }
it { is_expected.to eq %w[new-2 new-1] }
end