Database Design Historical Data Model - sql

Question

I am thinking a good design to capture history of product change. Suppose a user can have different products to trade for each day.
User Product Day
1 A 1
1 B 1
1 A 2
1 B 2
1 C 3
As we can see above on day 3, Product C is Added and Product A B are removed.
Thinking of below 2 design:
#1 Capture the product changes and store it as start and end date
User Product Start End
1 A 1 3
1 B 1 3
1 C 3 -
#2 Capture the product changes as 1 record
User Product Action Day
1 A Added 1
1 B Added 1
1 C Added 3
1 A Removed 3
1 B Removed 3
My following question is can these 2 models be converted to each other. For example, we can use Lead/Lag to convert #2 into #1.
Which design is better? Our system is using #2 to store the historical data.
Updated:
the intention to use the data is showing the product changes history.
For example, for a given date range, what's the product change for a particular user?

Answer 1

The second model seems better, at least if your main interest is in queries like "find all changes for all users and products, which occurred between DATE_1 and DATE_2".
With the second model, the query is trivial:
select * from (table) where (date) between DATE_1 and DATE_2;
How would you write the query for the first model?
Moreover, with the second model you could create an index on (user, date) - or even just on (date) - which will make quick work of the query. Even if you had indexes on the table in the first model, they wouldn't be used due to the complicated nature of the query.
While integrity constraints would be relatively difficult in both cases (as they are cross-rows), they would be much easier to implement - either with materialized views or with triggers - with the second model. In the first model you have to make sure there are no overlaps between the intervals. With the first model, if you partition by user and order by date, the condition is simply that the action alternates from row to row. Still not trivial to implement, but much simpler than the "non-overlapping intervals" condition for the first model.
To your other question: It is, indeed, trivial to go from either model to the other, using PIVOT and UNPIVOT. You do need an analytic function (ROW_NUMBER) before you PIVOT to go from model #2 to #1. You don't need any preparation to go from #1 to #2.

Answer 2

Personally, I think the first option is better. I'm assuming you have so many rows that the raw structure of a row per user, product and date is too heavy? Because for visualisations I think the raw table would work fine as is.
However, if you have to aggregate due to size, and do not need to know the amounts of the product nor how many users are selling them on any given day, then the first option would be easier to work with in my opinion just in terms of SQL. On the other hand, you will have a problem in case a product can have several start and end dates, since I am assuming a new entry would just overwrite the previous date stamp.
So, that in mind, I would personally create a table with a row per day(or monthly if you want to minimise the size of the table and monthly is granular enough for your use case). Then add a column for each product and whether or not they were sold that day. You could even do it with a count on the number of users selling that product, which would give you a little more detail. The only problem this model has, is that I would only use it in case it is truly static, historical data with no need to add new products.

Answer 3

You can convert from any one format to the other formats.
Data in the first format:
CREATE TABLE table1 (Usr, Product, Day) AS
SELECT 1, 'A', '1' FROM DUAL UNION ALL
SELECT 1, 'B', '1' FROM DUAL UNION ALL
SELECT 1, 'A', '2' FROM DUAL UNION ALL
SELECT 1, 'B', '2' FROM DUAL UNION ALL
SELECT 1, 'C', '3' FROM DUAL
Then:
SELECT usr,
product,
day + DECODE( action, 'Removed', 1, 0) AS day,
action
FROM (
SELECT Usr,
Product,
Day,
CASE
WHEN LAG( Day ) OVER ( PARTITION BY Usr, Product ORDER BY Day ) = Day - 1
THEN NULL
ELSE 'Added'
END AS Added,
CASE
WHEN LEAD( Day ) OVER ( PARTITION BY Usr, Product ORDER BY Day ) = Day + 1
THEN NULL
WHEN Day = MAX( Day ) OVER ()
THEN NULL
ELSE 'Removed'
END AS Removed
FROM table1
)
UNPIVOT ( action FOR value IN ( Added, Removed ) )
Outputs that data in the second form:
USR | PRODUCT | DAY | ACTION
--: | :------ | --: | :------
1 | A | 1 | Added
1 | A | 3 | Removed
1 | B | 1 | Added
1 | B | 3 | Removed
1 | C | 3 | Added
and:
SELECT Usr,
Product,
MIN( Day ) AS "Start",
CASE MAX( Day )
WHEN Last_Day
THEN NULL
ELSE MAX( Day ) + 1
END AS "End"
FROM (
SELECT Usr,
Product,
Day,
Day - ROW_NUMBER() OVER ( PARTITION BY Usr, Product ORDER BY Day ) AS grp,
MAX( Day ) OVER () AS last_day
FROM table1
)
GROUP BY Usr, Product, Grp, Last_Day
ORDER BY Usr, Product, "Start"
Outputs the data in the third format:
USR | PRODUCT | Start | End
--: | :------ | :---- | ---:
1 | A | 1 | 3
1 | B | 1 | 3
1 | C | 3 | null
Data in the second format:
CREATE TABLE table2 ( Usr, Product, Day, Action ) AS
SELECT 1, 'A', 1, 'Added' FROM DUAL UNION ALL
SELECT 1, 'A', 3, 'Removed' FROM DUAL UNION ALL
SELECT 1, 'B', 1, 'Added' FROM DUAL UNION ALL
SELECT 1, 'B', 3, 'Removed' FROM DUAL UNION ALL
SELECT 1, 'C', 3, 'Added' FROM DUAL;
Then you can convert it to the third format using:
SELECT Usr,
Product,
"Start",
"End"
FROM (
SELECT t.*,
ROW_NUMBER() OVER ( PARTITION BY Usr, Product, Action ORDER BY Day ) AS rn
FROM table2 t
)
PIVOT (
MAX( Day )
FOR Action IN (
'Added' AS "Start",
'Removed' AS "End"
)
)
Which outputs:
USR | PRODUCT | Start | End
--: | :------ | ----: | ---:
1 | A | 1 | 3
1 | B | 1 | 3
1 | C | 3 | null
Data in the third format:
CREATE TABLE table3 ( Usr, Product, "Start", "End" ) AS
SELECT 1, 'A', 1, 3 FROM DUAL UNION ALL
SELECT 1, 'B', 1, 3 FROM DUAL UNION ALL
SELECT 1, 'C', 3, NULL FROM DUAL;
Then to get the data in the first format you can use:
WITH unrolled_data ( Usr, Product, Day, "End" ) AS (
SELECT Usr, Product, "Start", "End"
FROM table3
UNION ALL
SELECT Usr, Product, Day + 1, "End"
FROM unrolled_data
WHERE Day + 1 < COALESCE( "End", 4 /* The last day + 1 */ )
)
SELECT Usr, Product, Day
FROM unrolled_data
ORDER BY Usr, Day, Product
Outputs:
USR | PRODUCT | DAY
--: | :------ | --:
1 | A | 1
1 | B | 1
1 | A | 2
1 | B | 2
1 | C | 3
And can convert to the second format using:
SELECT *
FROM table3
UNPIVOT ( Day FOR Action IN ( "Start" AS 'Added', "End" AS 'Removed' ) )
Which outputs:
USR | PRODUCT | ACTION | DAY
--: | :------ | :------ | --:
1 | A | Added | 1
1 | A | Removed | 3
1 | B | Added | 1
1 | B | Removed | 3
1 | C | Added | 3
(and you can combine queries to convert from 2-to-1.)
db<>fiddle here