How to optimize multiple left-joins SQL SELECT-query? - sql

Question

Situation:
We have a database "base1" ~ 6 million lines of data, which shows the actual customer purchases and the day of purchase + the parameters of this purchase.
CREATE TABLE base1 (
User_id NOT NULL PRIMARY KEY ,
PurchaseDate date,
Parameter1 int,
Parameter2 int,
...
ParameterK int );
And also another database "base2" ~ 90 million lines of data, which actually shows the same thing, but instead of the day of purchase, a weekly section is used (for example: all weeks for 4 years for each client - if there was no purchase for N week, the client is still shown).
CREATE TABLE base2 (
Users_id NOT NULL PRIMARY KEY ,
Week_start date ,
Week_end date,
Parameter1 int,
Parameter2 int,
...
ParameterN int );
The task to do the following query:
-- a = base1 , b , wb%% = base2
--create index idx_uid_purch_date on base1(Users_ID,Purchasedate);
SELECT a.Users_id
-- Checking whether the client will make a purchase in next week and the purchase will be bought on condition
,iif(b.Users_id is not null,1,0) as User_will_buy_next_week
,iif(b.Users_id is not null and b.Parameter1 = 1,1,0) as User_will_buy_on_Condition1
-- about 12 similar iif-conditions
,iif(b.Users_id is not null and (b.Parameter1 = 1 and b.Parameter12 = 1),1,0)
as User_will_buy_on_Condition13
-- checking on the fact of purchase in the past month, 2 months ago, 2.5 months, etc.
,iif(wb1m.Users_id is null,0,1) as was_buy_1_month_ago
,iif(wb2m.Users_id is null,0,1) as was_buy_2_month_ago
,iif(wb25m.Users_id is null,0,1) as was_buy_25_month_ago
,iif(wb3m.Users_id is null,0,1) as was_buy_3_month_ago
,iif(wb6m.Users_id is null,0,1) as was_buy_6_month_ago
,iif(wb1y.Users_id is null,0,1) as was_buy_1_year_ago
,a.[Week_start]
,a.[Week_end]
into base3
FROM base2 a
-- Join for User_will_buy
left join base1 b
on a.Users_id =b.Users_id and
cast(b.[PurchaseDate] as date)>=DATEADD(dd,7,cast(a.[Week_end] as date))
and cast(b.[PurchaseDate] as date)<=DATEADD(dd,14,cast(a.[Week_end] as date))
-- Joins for was_buy
left join base1 wb1m
on a.Users_id =wb1m.Users_id
and cast(wb1m.[PurchaseDate] as date)>=DATEADD(dd,-30-4,cast(a.[Week_end] as date))
and cast(wb1m.[PurchaseDate] as date)<=DATEADD(dd,-30+4,cast(a.[Week_end] as date))
/* 4 more similar joins where different values are added in
DATEADD (dd, %%, cast (a. [Week_end] as date))
to check on the fact of purchase for a certain period */
left outer join base1 wb1y
on a.Users_id =wb1y.Users_id and
cast(wb1y.[PurchaseDate] as date)>=DATEADD(dd,-365-4,cast(a.[Week_end] as date))
and cast(wb1y.[PurchaseDate] as date)<=DATEADD(dd,-365+5,cast(a.[Week_end] as date))
Because of the huge number of Joins and rather big databases - this script runs for about 24 hours, which is incredibly long.
Main time, as the execution plan shows, is spent on "Merge Join" and view the rows of the table from base1 and base2, and insert the data into another base3 table.
Question: Is it possible to optimize this query so it works faster?
Perhaps using one Join instead or something.
Help please, I'm not that smart enough :(
Thanx everybody for your answers!
UPD: Maybe use of different type of joins (merge, loop, or hash) may help me, but can't really check this theory. Maybe someone can tell me whether it's right or wrong ;)

Answer 1

You want to have all 90 million base2 rows in your result, each with additional information on base1 data. So, what the DBMS must do is a full table scan on base2 and quickly find related rows in base1.
The query with EXISTS clauses would look something like this:
select
b2.users_id,
b2.week_start,
b2.week_end,
case when exists
(
select *
from base1 b1
where b1.users_id = b2.users_id
and b1.purchasedate between dateadd(day, 7, cast(b2.week_end as date))
and dateadd(day, 14, cast(b2.week_end as date))´
) then 1 else 0 end as user_will_buy_next_week,
case when exists
(
select *
from base1 b1
where b1.users_id = b2.users_id
and b1.parameter1 = 1
and b1.purchasedate between dateadd(day, 7, cast(b2.week_end as date))
and dateadd(day, 14, cast(b2.week_end as date))´
) then 1 else 0 end as user_will_buy_on_condition1,
case when exists
(
select *
from base1 b1
where b1.users_id = b2.users_id
and b1.parameter1 = 1
and b1.parameter2 = 1
and b1.purchasedate between dateadd(day, 7, cast(b2.week_end as date))
and dateadd(day, 14, cast(b2.week_end as date))´
) then 1 else 0 end as user_will_buy_on_condition13,
case when exists
(
select *
from base1 b1
where b1.users_id = b2.users_id
and b1.purchasedate between dateadd(day, -30-4, cast(b2.week_end as date))
and dateadd(day, -30+4, cast(b2.week_end as date))´
) then 1 else 0 end as was_buy_1_month_ago,
...
from base2 b2;
We can easily see that this will take a long time, because all conditions must be checked per base2 row. That is 9 million times 7 lookups. The only thing we can do about this is to provide an index, hoping the query will benefit from it.
create index idx1 on base1 (users_id, purchasedate, parameter1, parameter2);
We can add more indexes, so the DBMS can choose between them on selectivity. Later we can check whether they are used and drop them in case they aren't.
create index idx2 on base1 (users_id, parameter1, purchasedate);
create index idx3 on base1 (users_id, parameter1, parameter2, purchasedate);
create index idx4 on base1 (users_id, parameter2, parameter1, purchasedate);

Answer 2

I assume that the base1 table stores information about the current week purchases.
If that is true, in the query conditions of joins you could ignore [PurchaseDate] parameter, replacing it with the current date constant instead. In that case your DATEADD functions will be applied to the current date and will be constants in the conditions of joins:
left join base1 b
on a.Users_id =b.Users_id and
DATEADD(day,-7,GETDATE())>=a.[Week_end]
and DATEADD(day,-14,GETDATE())<=a.[Week_end]
To have the query above running correctly you should limit b.[PurchaseDate] to the current day.
Then you could run another query, for the purchases made yesterday, and all DATEADD constants in join conditions corrected by -1
And so on, up to 7 queries, or whatever timespan the base1 table covers.
You could also implement grouping of [PurchaseDate] values by days, recalculate constants and make all of that in a single query, but I'm not ready to spend time creating it myself. :)

Answer 3

If you have recurring argument such as DATEADD(dd,-30-4,cast(a.[Week_end] as date)) for example, to make it SARGable you can create an index on it (SQL Server can't). Postgres can do this:
create index ix_base2__34_days_ago on base2(DATEADD(dd,-30-4, cast([Week_end] as date)))
Then an expression like the following would be SARGable as index on DATEADD(dd,-30-4, cast([Week_end])) would be utilized by your database, hence a condition like the following will be fast if you have an index like on the example above.
and cast(wb1m.[PurchaseDate] as date) >= DATEADD(dd,-30-4,cast(a.[Week_end] as date))
Note that casting PurchaseDate to date yields a SARGable expression, despite cast looking like a function, as SQL Server has special handling of datetime to date, an index on datetime field is SARGable even you search on datetime field partially (the date part only). Similar to partial expression like, where lastname LIKE 'Mc%', that expression is SARGable even if an index is for the whole lastname field. I digress.
To somewhat achieve the index on expression on SQL Server, you can create a computed column on that expression.., e.g.,
CREATE TABLE base2 (
Users_id NOT NULL PRIMARY KEY ,
Week_start date ,
Week_end date,
Parameter1 int,
Parameter2 int,
Thirty4DaysAgo as DATEADD(dd,-30-4, cast([Week_end] as date))
)
..and then create index on that column:
create index ix_base2_34_days_ago on base2(Thirty4DaysAgo)
Then change your expression to:
and cast(wb1m.[PurchaseDate] as date) >= a.Thirty4DaysAgo
That's what I would recommend before, change the old expression to use the computed column. However, upon further searching, it looks like you can just retain your original code, as SQL Server can intelligently match an expression to the computed column, and if you have an index on that column, your expression would be SARGable. Thus your DBA can optimize things behind the scenes and your original code would run optimized without requiring any changes on your code. So no need to change the following, and it will be SARGable (granted that your DBA created a computed column for dateadd(recurring parameters here) expression, and applied index on it) :
and cast(wb1m.[PurchaseDate] as date) >= DATEADD(dd,-30-4,cast(a.[Week_end] as date))
The only downside (when compared to Postgres) is you still have the dangling computed column on your table when using SQL Server :)
Good read: https://littlekendra.com/2016/03/01/sql-servers-year-function-and-index-performance/