SQL Schema - car model with modifiers as unique - sql

Question

I need to build a DB for the following scenario:
I will have an input stream of auctions, and I want to make a price histogram for items on said auction (ie. what they usually go for etc).
The input stream looks something like:
{['item_id': 1, ... 'price': 123, ...],
['item_id': 1, ... 'price': 124, ... modifiers: [1, 2, 3],
['item_id': 1, ... 'price': 125, ... modifiers: [100, 150, 500...],
['item_id': 2, ... 'price': 200, ...],
...}
As you might have noticed, item doesn't only consist of some id, but also of modifiers. Think of it as a car that can be modified with extra stuff (e.g. AC, electronic windows etc).
What would be the most efficient way to store this information? Basically what I want to have is a unique id for each combination that can occur. It's not necessary to store it at all times, but if there is an auction for such an combination, and the combination doesn't exist yet, create it then.
I thought of something like:
base_item:
id
modifier:
id
item:
id (autonumber)
base_item_id
item_modifications:
item_id (FK item.id)
modification_id (FK modifier.id)
item_price_history:
item_id (FK item.id)
price
time
This setup might work. The problem is, imagine I have hundreds of millions of such auctions every day (ie. the auction's information is updated every 20 minutes and it cosists of 2 million auctions in average).
I want to be able to quickly do something like: INSERT INTO item_price_history VALUES (some_item_id, some_price, now()) but in order to do that, I need to find some_item_id. I know base_item_id and modifiers (from auction itself), but doing such call hundreds of millions of times is quite costly I think?
Ie, pseudo code:
for a in auctions:
base_item_id = a['item_id']
modifiers = a['modifiers']
price = a['price']
actual_item_id = some_query(base_item_id, modifiers) #expensive. Can be avoided?
insert_into_histogram(auctual_item_id, price) #expensive but necessary I think
Is there some obvious mistake I'm making in this design?

Answer 1

The schema you describe is the textbook solution.
But wow, that would be a beast to work with. As I understand it, every time you added a price record, you would have to find the item record with that exact set of parameters: no more, no less. And if no such item record existed, you would then have to create the item record. Only then could you add the price record.
While I think one should be very careful about denormalizing, I'd be sorely tempted to denormalize in this case. Namely, it seems to me that in practice, the key to an item record is the combination of the base item id plus the modifiers. I'd be tempted to to create a "modifier string" formed by stringing together codes or IDs for all the modifiers. Of course to be workable they'd have to be strung together in a defined sequence, like you can't have both "1,2" and "2,1". But then you could easily find the desired item record: just have a function that builds the concatenated modifier string, and select item where base_item_id=#base and modifiers=#modifiers. If not found, create the record and all the associated modifier records.
I'd be strongly inclined to make this modifier string be redundant with individual modifier records, but data that is strung together like this is very difficult to process. I mean, if you have a textbook schema like you describe, and someone wants to know prices for cars with air conditioning, it's very easy to select * from price where price.item in (select id from item join modifier on modifier.item_id=item.id where modifier.name='AC'). But try and do that on the concatenated string, say the ID for AC is "17". select blah blah where modifier_string like '%17%' doesn't work: it will find 117 and 171 and so on. like '%,17,%' doesn't work because it won't find it if it's the first or the last. Etc. That's why I routinely tell people NOT to string data together like this in general: create separate records. But if the most common use case is that you want the record with a specific combination of modifiers, creating a redundant modifier string is a plausible denormalization. (And the first time I typed that I accidentally typed 'demoralization', which may have been a Freudian slip.)