I have a table which has a simple parent child structure
products:
- id
- product_id
- time_created
- ... a few other columns
It is a parent if product_id IS NULL. Product id behaves here like parent_id. Data inside looks like this:
id | product_id
1 NULL
2 1
3 1
4 NULL
4 4
This table is updated every night a new versions are added.
Every user is using a lot of these products but only one version. User is notified if new rows are added for an product_id.
He can stop using id:2 and start using id:3. An another user will continue using id:2 etc.
products table is updated every night and it grows pretty fast. There are around 500000 rows at the moment and every night adds around 20000, probably 5-7000000 changes (new rows) per year.
Is there a way to optimize this database/table structure? Should I change anything? Is it a problem to have so much data in one table?
Your question is not clear. The sample data is suggesting that the parent-child relationship is only one level deep. If so, this is not a particularly hard problem. You can create a query to look up the most recent product id for each product -- and I'm assuming this is the one with the maximum id:
select id, product_id,
max(id) over (partition by coalsesce(product_id, id)) as biggest_id
from table t;
This is then a lookup table, to get the biggest id. It would produce:
id | product_id | biggest_id
1 NULL 3
2 1 3
3 1 3
4 NULL 4
4 4 4
If your table has deeper hierarchies, you can solve the problem using recursive CTEs, or by doing the calculation when the table is updated.
Related
I have relational category & product tables. Categories are hierarchical. I will have queries based on category, for example
select *
from products
where CatId = 3
or
select *
from products
where CatId = 1
I have 6 level of category and 24 million row for products, I have to find fast and optimal solutions. My question is which structure is suitable.
I write some options, feel free to suggest a better alternative.
Current category table:
Id ParentId Name
---------------------
1 null CatA
2 null CatB
3 1 CatAa
4 2 CatBa
Product table option 1
Id Cat Name
------------------
1 3 Product_1
2 4 Product_2
Product table option 2
Id CatLevel1 CatLevel2 ... Name
-------------------------------------
1 1 3 . Product_1
2 2 4 . Product_2
Product table option 3
Id Cats Name
------------------
1 1:3 Product_1
2 2:4 Product_2
Always keep option one, plus some denormalised tables (options two onwards) if you so desire. By keeping option one, you have the source truth to revert to or derive the others from.
Option two is only recommended if the searcher always knows what depth/level to search at. For example, if they know they need Level2=CATAb then it works, but if they don't know CATAb is at level two, they don't know which column to look in. It also relies on knowing how many levels to represent; if you can have a hundred levels, you need a hundred columns, and it's fragile of you need to add more depths. Generally, this doesn't apply and so is generally not a good optimisation.
Option three is a straight no. Never store multiple values in a one field (one column of one row). It will make Efficient searching of that column next to impossible.
The alternative to option three is to have a "link" table. Just two columns, category_id and product_id. Then you list all ancestors of a product, just on different rows.
category_id
product_id
1
1
3
1
2
2
4
2
These are all known as adjacency lists. A different model altogether is Nested Sets. I'm on my phone, and it's hard to describe without lots of formatting, but if you research online you'll find lots of information. They're much harder to comprehend and implement Initially, but very fast at retrieval when specifying a parent.
Your product table option 1 is fine and need no change
product_id,
category_id,
... other attributes
Your problem is in accessing the product based on the category hierarchy - which would make a need of a hierarchical query to get all categories in the tree below your selected category.
Instead of
select * from product where category_id = 1;
you'll need to write an additional hierarchical query to get the whole hierarchy tree
with cat_tree (id) as (
select id
from category where id = 1
UNION ALL
select ca.id
from cat_tree ct
join category ca
on ct.id = ca.parent_id
)
select * from product
where category_id in
(select id from cat_tree);
Which may not be practicable, but you may simplify it by denormalizing the category table
Let's assume your category data is such as
ID PARENT_ID
---------- ----------
1
3 1
5 3
6 3
The query below, which may be implemented as a MATERIALIZED VIEW that is refreshed on each category change pre-calculates all direct and indirect parent and child relations.
The result is
ID CHILD_ID
---------- ----------
1 1
1 3
1 5
1 6
3 3
3 5
3 6
5 5
6 6
E.g. for 1 you get itself, all its child's, their child's etc.
Using this category_denormobject your query can be simplified to
select *
from product
where category_id in
(select child_id from category_denorm where id = 1);
I am trying to diagnose problems with a larger query, so I started from the basics. I noticed that when I query one table I am getting 12 rows selected but the last row has it's own header, which is the same header of the result set. Why would this be?
SELECT *
FROM testtable;
Results
testid personid role
__________________________
1 1 memeber
2 3 memeber
3 1 chair
etc etc etc
then at the last row I get another header
testid personid role
____________________________
4 2 member
This may be a dumb question but I couldn't find an answer searching and I am new and learning SQL.
Here is a theoretical scenario,
Suppose I have a client table and an invoice table.
1 client can have many invoices.
Now I want each invoice to have an invoice number that is unique to that client
i.e.
ClientId InvoiceNo
1 IN0001
2 IN0001
2 IN0002
2 IN0003
3 IN0001
Currently I am controlling this in my application by looking at max values etc but this is obviously not a great solution. I would much rather get my database to do this for me, as it should remove the risk of creating duplicate invoice numbers for a single client (race condition?)
I have been reading up on Sql Server 2012's SEQUENCE which sounds great, but the problem is that I would still need a seperate sequence per client
i.e.
CREATE SEQUENCE InvoiceNum_Client1.....
CREATE SEQUENCE InvoiceNum_Client2.....
CREATE SEQUENCE InvoiceNum_Client3.....
but something feels very dirty about making db meta changes from my app everytime a new client is created. Also then my trigger would have to do something like this (which I wouldn't even begin to know how to do)
NEXT VALUE FOR InvoiceNum_Client + #ClientId
etc
So my next thought was to have a "sequence" table, i.e.
ClientID INSequenceNumber
1 1
2 3
3 3
And in my trigger grab the InSequenceNumber for a given client, use it to make my invoiceNumber, and then update the sequence table, incrementing InSequenceNumber by 1 for the same client. It should have the same effect, but I am just not sure about the inner workings of transactions and scope etc
So my questions are
Is there any big disadvantage to my self rolled sequence method?
Is there another solution that I am perhaps overlooking?
Thanks!
Is there a specific reason for requiring that the invoice numbers are only unique per client? In most ERP systems, invoice numbers are typically globally unique, making implementation much easier. No matter what, you should have an Invoice table that contains a primary key (and you shouldn't use composite primary keys - that's just downright bad data modelling).
This leaves us with a scenario where you might not need to store the "per-client-invoice-number" in the database at all. Assuming you have a table called "Invoices" containing the following data:
Id | ClientId
---------------
1 | 1
2 | 1
3 | 2
4 | 1
5 | 3
6 | 2
Here, Id is the Primary Key of the Invoices table, and ClientId is a foreign key. A query like this:
SELECT
ClientId,
'IN' + RIGHT('0000' +
CONVERT(VARCHAR, ROW_NUMBER() OVER (PARTITION BY ClientId
ORDER BY Id)) AS InvoiceNo,
Id
FROM Invoices
ORDER BY ClientId, InvoiceNo
Would return:
ClientId | InvoiceNo | Id
---------------------------
1 | IN0001 | 1
1 | IN0002 | 2
1 | IN0003 | 4
2 | IN0001 | 3
2 | IN0002 | 6
3 | IN0001 | 5
Why do the clients have to have their own sequences? Have a global sequence number. Then, if you want to get the client sequences in order, use row_number():
select i.*, row_number() over (partition by clientid order by invoiceno) as ClientInvoiceSequence
from invoices;
Note: you might want the order by to be by another field such as date.
If you start storing this information in the database, you will need to do a lot of bookkeeping and careful transaction management:
What happens when two invoices are entered "at the same time"?
What happens when an invoice is deleted?
What happens when an invoice is modified in such a way that it might change the sequence?
You are much better off using an identity column and calculating the sequence when you need it.
Given the OrderLine table below:
OrderID OrderLineID
======= ===========
1 1
1 2
2 3
3 4
1 5
3 6
... ...
... ...
221 123 365 282
What is the most efficient way to find the FIRST OrderLine for each order, given that this information is required to access every now and then by the user?
This is my SQL to find the first OrderLine, but it takes about 3~5 seconds to execute every-time. (about 300k rows)
SELECT OrderID, MIN(OrderLineID)
FROM OrderLine
GROUP BY OrderID
It's very expensive to repeat this every-time when I need to find the first orderline to join with another table. Consider that changing the table structure is not an option, what possible solution do I have to improve this?
Try adding an index by OrderID and OrderLineID.
(You say you can't change the table structure. If you were allowed to change the structure, you could add a flag that identifies the first line of every order, then index by that flag.)
I have three tables: store, product, storeproduct.
It doesn't really matter what's in the store and the product table, just know there is a storeID in the store table, and a productID in the product table. However the storeproduct table keeps track of the different products each store has. So the storeproduct table has two columns. The storeID column, and the productID column, both foreign keys from the store and the product table.
Is there a way to put a constraint or check on any of the table to make sure that a store must have more than 0 products, and less than 50 products.
Note: I do not want a select statement to do this. I just want to know if there is a way to put a constraint or a check when creating the tables.
The point of this is so a user cannot insert into the storeproduct table if there are already 50 products(rows) with the same storeID, or delete from the storeproduct table if deleting a row will cause the last row with that storeID to be gone.
The storeproduct table might look like this
storeID productID
1 1
1 2
1 3
2 4
2 5
2 6
2 7
3 4
3 2
3 6
3 1
3 8
Actually, depending on your database you may be able to do this.
Oracle (and maybe others) provide materialized views which you can apply constraints to. So you could create the MV with a column PRODUCTS_IN_STORES (being something like select storeID, count(*) as PRODUCTS_IN_STORES from stores left outer join storeproduct on store.storeid=storeproduct.storeid group by store.storeid .Then put a constraint on it asserting that PRODUCTS_IN_STORES is between 0 and 50 or whatever.
http://www.sqlsnippets.com/en/topic-12896.html
and
http://asktom.oracle.com/pls/asktom/f?p=100:11:0::::P11_QUESTION_ID:21389386132607
Not a complete answer for you, but something to think about and hopefully set you on your way.