I have a table, horribly designed (not my doing thankfully), that stores data in a fashion similar to the following:
[key], [lease_id], [building_name], ~20 more columns of data
A lease_id can and will exist for a centre as well as head office. I've been asked to find all instances where data in a building for a lease doesn't match data in head office for the same lease.
I can do this, quite easily, with a self join. The challenge here is that there are about 20 columns to compare and although I could type each one in manually I was wondering if there's a better way to do this (which would also mean the query can be used in future, accounting for any table changes).
In syntaxtically ridiculous psuedo code- I want to do something similar to what the following would do if it were to work:
select lp.*
from lease_proposal lp
inner join
(
select *
from lease_proposal lp2
where building_id = '001' -- assume 001 is head office for sake of example
) lp2
on lp2.lease_id = lp.lease_id
where lp.* <> lp2.*
You could do an INTERSECT operation to find all rows where all data matched, then LEFT JOIN that result and select only the rows where there wasn't an intersection:
SELECT
a.*
FROM
lease_proposal a
LEFT JOIN
(
SELECT *
FROM lease_proposal
INTERSECT
SELECT *
FROM lease_proposal
WHERE building_id = 001
) b ON a.lease_id = b.lease_id
WHERE
b.lease_id IS NULL
If SQL Server supported it, you could also use a NATURAL LEFT JOIN like so:
SELECT
a.*
FROM
lease_proposal a
NATURAL LEFT JOIN
(
SELECT *
FROM lease_proposal
WHERE building_id = 001
) b
WHERE b.lease_id IS NULL
Related
I have several databases (nobu and bu) with exact same tables (one is just a back up of the other).
I need to get values from a table from both databases to join them with other tables then I obviously use an UNION. The thing is, some products have different names in the tables from both bu and nobu.
I then tried to select only one database about this table (I used nobu since it's the latest one), but I noticed that some products are not in nobu, but are actually in bu (which makes it not a backup anymore).
The part of the query in which I need this looks like this :
With this I get duplicates
... INNER JOIN (SELECT * FROM nobu.dbo.product UNION SELECT * FROM bu.dbo.product) AS product
ON [...] INNER JOIN (SELECT * FROM nobu.dbo.name UNION SELECT bu.dbo.name) AS name
ON product.key = name.id ...
With this I get some of the products with NULL name since it doesn't exist on nobu
... INNER JOIN (SELECT * FROM nobu.dbo.product UNION SELECT * FROM bu.dbo.product) AS product
ON [...] INNER JOIN (SELECT * FROM nobu.dbo.name) AS name
ON product.key = name.id ...
I wanted to know if there is a way to perform a LEFT UNION or something like that, to get all the values from nobu, and if there is no data, take the ones from bu, without getting the duplicates (since they can have different names on both databases).
If only names have been changed and suggesting that table names is not a big table and will not create performance issues then this code below will do the job:
INNER JOIN (SELECT * FROM nobu.dbo.product UNION SELECT * FROM bu.dbo.product) AS product
ON [...] INNER JOIN (SELECT * FROM nobu.dbo.name UNION SELECT bu.dbo.name WHERE id NOT IN (SELECT id FROM nobu.dbo.name)) AS name
ON product.key = name.id
Can someone help explain to me how when I have 12 rows in table A and 10 in B and I do an inner join , I would get more rows than
in both A and B ?
Same with left and right joins...
This is just a simplified example. Let me share one of my issues with you
I have 2 views ; which was originally SQL on 2 base tables Culture and Trials.
And then when attempting to add another table Culture Steps, one of the team members separated the SQL into 2 views
Since this produces an error when updating(modification cannot be done as it affects multiple base tables), I would like to get
back to changing the SQL such that I no longer use the views but achieve the same results.
One of the views has
SELECT some columns
FROM dbo.Culture RIGHT JOIN
dbo.Trial ON dbo.Culture.cultureID = dbo.Trial.CultureID LEFT OUTER JOIN
dbo.TrialCultureSteps_view_part1 ON dbo.Culture.cultureID = dbo.TrialCultureSteps_view_part1.cultureID
The other TrialCultureSteps_view_part1 view
SELECT DISTINCT dbo.Culture.cultureID,
(SELECT TOP (1) WeekNr
FROM dbo.CultureStep
WHERE (CultureID = dbo.Culture.cultureID)
ORDER BY CultureStepID) AS normalstartweek
FROM dbo.Culture INNER JOIN
dbo.CultureStep AS CultureStep_1 ON dbo.Culture.cultureID = CultureStep_1.CultureID
So how can I combine the joins the achieve the same results using SQL only on tables without the need for views?
Welcome to StackOverflow! This link might be a good place to start in your understanding of JOINs. Essentially, the 'problem' you describe boils down to the fact that one or more of your sources (Trial, Culture, or the TrialCultureSteps view) has more than one record per CultureID - in other words, the same CultureID (#1) shows up on multiple rows.
Based solely on that ID, I'd execute the following three queries. Anything that is returned by them is the 'cause' of your duplications - the culture ID shows up more than once, so you'll have to JOIN on more than just CultureID. If, as I half-suspect, your view is the one that has multiple Culture IDs, you'll need to modify it to only return one record, or change the way that you JOIN to it.
SELECT *
FROM Trial
WHERE CultureID IN
(
SELECT CultureID
FROM Trial
GROUP BY CultureID
HAVING COUNT(*) > 1
)
ORDER BY CultureID
SELECT *
FROM Culture
WHERE CultureID IN
(
SELECT CultureID
FROM Culture
GROUP BY CultureID
HAVING COUNT(*) > 1
)
ORDER BY CultureID
SELECT *
FROM TrialCultureSteps_view_part1
WHERE CultureID IN
(
SELECT CultureID
FROM TrialCultureSteps_view_part1
GROUP BY CultureID
HAVING COUNT(*) > 1
)
ORDER BY CultureID
Let me know if any of these return values!
The comments explain the JOIN issues. As for rewriting, any views could be replaced with CTEs.
One other way to rewrite the query, would be : (Though having sample data and expected result would make this easier to confirm that it's correct)
;with TrialCultureSteps_view_part1 AS
(
Select Row_number() OVER (Partition BY CultureID ORDER BY CultureStepID) RowNumber
, WeekNr
, CultureID
)
SELECT some columns
dbo.trial LEFT OUTER JOIN
dbo.Culture ON dbo.Culture.cultureID = dbo.Trial.CultureID LEFT OUTER JOIN
TrialCultureSteps_view_part1 ON dbo.Culture.cultureID = dbo.TrialCultureSteps_view_part1.cultureID and RowNumber=1
Access code, I'm less familiar with the syntax, but I know that Row_Number() isn't available and I don't believe it has CTE syntax either. So, we'd need to put in some more nested derived tables.
SELECT some columns
dbo.trial LEFT OUTER JOIN
dbo.Culture ON dbo.Culture.cultureID = dbo.Trial.CultureID LEFT OUTER JOIN
( Select cs.CultureID, cs.WeekNr FROM
( SELECT CultureID, MIN(CultureStepID) CultureStepID
FROM dbo.CultureStep
GROUP BY CultureID
) Fcs INNER JOIN
CultureStep cs ON fcs.cultureStepID=cs.CultureStepID
) TrialCultureSteps_view_part1 ON dbo.Culture.cultureID = TrialCultureSteps_view_part1.cultureID
Assumptions here, is that CultureStepID is a PK for CultureStep. No assumption that a step must exist for each Culture entry.
I am trying to filter a single table (master) by the values in multiple other tables (filter1, filter2, filter3 ... filterN) using only joins.
I want the following rules to apply:
(A) If one or more rows exist in a filter table, then include only those rows from the master that match the values in the filter table.
(B) If no rows exist in a filter table, then ignore it and return all the rows from the master table.
(C) This solution should work for N filter tables in combination.
(D) Static SQL using JOIN syntax only, no Dynamic SQL.
I'm really trying to get rid of dynamic SQL wherever possible, and this is one of those places I truly think it's possible, but just can't quite figure it out. Note: I have solved this using Dynamic SQL already, and it was fairly easy, but not particularly efficient or elegant.
What I have tried:
Various INNER JOINS between master and filter tables - works for (A) but fails on (B) because the join removes all records from the master (left) side when the filter (right) side has no rows.
LEFT JOINS - Always returns all records from the master (left) side. This fails (A) when some filter tables have records and some do not.
What I really need:
It seems like what I need is to be able to INNER JOIN on each filter table that has 1 or more rows and LEFT JOIN (or not JOIN at all) on each filter table that is empty.
My question: How would I accomplish this without resorting to Dynamic SQL?
In SQL Server 2005+ you could try this:
WITH
filter1 AS (
SELECT DISTINCT
m.ID,
HasMatched = CASE WHEN f.ID IS NULL THEN 0 ELSE 1 END,
AllHasMatched = MAX(CASE WHEN f.ID IS NULL THEN 0 ELSE 1 END) OVER ()
FROM masterdata m
LEFT JOIN filtertable1 f ON join_condition
),
filter2 AS (
SELECT DISTINCT
m.ID,
HasMatched = CASE WHEN f.ID IS NULL THEN 0 ELSE 1 END,
AllHasMatched = MAX(CASE WHEN f.ID IS NULL THEN 0 ELSE 1 END) OVER ()
FROM masterdata m
LEFT JOIN filtertable2 f ON join_condition
),
…
SELECT m.*
FROM masterdata m
INNER JOIN filter1 f1 ON m.ID = f1.ID AND f1.HasMatched = f1.AllHasMatched
INNER JOIN filter2 f2 ON m.ID = f2.ID AND f2.HasMatched = f2.AllHasMatched
…
My understanding is, filter tables without any matches simply must not affect the resulting set. The output should only consist of those masterdata rows that have matched all the filters where matches have taken place.
SELECT *
FROM master_table mt
WHERE (0 = (select count(*) from filter_table_1)
OR mt.id IN (select id from filter_table_1)
AND (0 = (select count(*) from filter_table_2)
OR mt.id IN (select id from filter_table_2)
AND (0 = (select count(*) from filter_table_3)
OR mt.id IN (select id from filter_table_3)
Be warned that this could be inefficient in practice. Unless you have a specific reason to kill your existing, working, solution, I would keep it.
Do inner join to get results for (A) only and do left join to get results for (B) only (you will have to put something like this in the where clause: filterN.column is null) combine results from inner join and left join with UNION.
Left Outer Join - gives you the MISSING entries in master table ....
SELECT * FROM MASTER M
INNER JOIN APPRENTICE A ON A.PK = M.PK
LEFT OUTER JOIN FOREIGN F ON F.FK = M.PK
If FOREIGN has keys that is not a part of MASTER you will have "null columns" where the slots are missing
I think that is what you looking for ...
Mike
First off, it is impossible to have "N number of Joins" or "N number of filters" without resorting to dynamic SQL. The SQL language was not designed for dynamic determination of the entities against which you are querying.
Second, one way to accomplish what you want (but would be built dynamically) would be something along the lines of:
Select ...
From master
Where Exists (
Select 1
From filter_1
Where filter_1 = master.col1
Union All
Select 1
From ( Select 1 )
Where Not Exists (
Select 1
From filter_1
)
Intersect
Select 1
From filter_2
Where filter_2 = master.col2
Union All
Select 1
From ( Select 1 )
Where Not Exists (
Select 1
From filter_2
)
...
Intersect
Select 1
From filter_N
Where filter_N = master.colN
Union All
Select 1
From ( Select 1 )
Where Not Exists (
Select 1
From filter_N
)
)
I have previously posted a - now deleted - answer based on wrong assumptions on you problems.
But I think you could go for a solution where you split your initial search problem into a matter of constructing the set of ids from the master table, and then select the data joining on that set of ids. Here I naturally assume you have a kind of ID on your master table. The filter tables contains the filter values only. This could then be combined into the statement below, where each SELECT in the eligble subset provides a set of master ids, these are unioned to avoid duplicates and that set of ids are joined to the table with data.
SELECT * FROM tblData INNER JOIN
(
SELECT id FROM tblData td
INNER JOIN fa on fa.a = td.a
UNION
SELECT id FROM tblData td
INNER JOIN fb on fb.b = td.b
UNION
SELECT id FROM tblData td
INNER JOIN fc on fc.c = td.c
) eligible ON eligible.id = tblData.id
The test has been made against the tables and values shown below. These are just an appendix.
CREATE TABLE tblData (id int not null primary key identity(1,1), a varchar(40), b datetime, c int)
CREATE TABLE fa (a varchar(40) not null primary key)
CREATE TABLE fb (b datetime not null primary key)
CREATE TABLE fc (c int not null primary key)
Since you have filter tables, I am assuming that these tables are probably dynamically populated from a front-end. This would mean that you have these tables as #temp_table (or even a materialized table, doesn't matter really) in your script before filtering on the master data table.
Personally, I use the below code bit for filtering dynamically without using dynamic SQL.
SELECT *
FROM [masterdata] [m]
INNER JOIN
[filter_table_1] [f1]
ON
[m].[filter_column_1] = ISNULL(NULLIF([f1].[filter_column_1], ''), [m].[filter_column_1])
As you can see, the code NULLs the JOIN condition if the column value is a blank record in the filter table. However, the gist in this is that you will have to actively populate the column value to blank in case you do not have any filter records on which you want to curtail the total set of the master data. Once you have populated the filter table with a blank, the JOIN condition NULLs in those cases and instead joins on itself with the same column from the master data table. This should work for all the cases you mentioned in your question.
I have found this bit of code to be faster in terms of performance.
Hope this helps. Please let me know in the comments.
SELECT C.* FROM StockToCategory STC
INNER JOIN Category C ON STC.CategoryID = C.CategoryID
WHERE STC.StockID = #StockID
VS
SELECT * FROM Category
WHERE CategoryID IN
(SELECT CategoryID FROM StockToCategory WHERE StockID = #StockID)
Which is considered the correct (syntactically) and most performant approach and why?
The syntax in the latter example seems more logical to me but my assumption is the JOIN will be faster.
I have looked at the query plans and havent been able to decipher anything from them.
Query Plan 1
Query Plan 2
The two syntaxes serve different purposes. Using the Join syntax presumes you want something from both the StockToCategory and Category table. If there are multiple entries in the StockToCategory table for each category, the Category table values will be repeated.
Using the IN function presumes that you want only items from the Category whose ID meets some criteria. If a given CategoryId (assuming it is the PK of the Category table) exists multiple times in the StockToCategory table, it will only be returned once.
In your exact example, they will produce the same output however IMO, the later syntax makes your intent (only wanting categories), clearer.
Btw, yet a third syntax which is similar to using the IN function:
Select ...
From Category
Where Exists (
Select 1
From StockToCategory
Where StockToCategory.CategoryId = Category.CategoryId
And StockToCategory.Stock = #StockId
)
Syntactically (semantically too) these are both correct. In terms of performance they are effectively equivalent, in fact I would expect SQL Server to generate the exact same physical plans for these two queries.
T think There are just two ways to specify the same desired result.
for sqlite
table device_group_folders contains 10 records
table device_groups contains ~100000 records
INNER JOIN: 31 ms
WITH RECURSIVE select_childs(uuid) AS (
SELECT uuid FROM device_group_folders WHERE uuid = '000B:653D1D5D:00000003'
UNION ALL
SELECT device_group_folders.uuid FROM device_group_folders INNER JOIN select_childs ON parent = select_childs.uuid
) SELECT device_groups.uuid FROM select_childs INNER JOIN device_groups ON device_groups.parent = select_childs.uuid;
WHERE 31 ms
WITH RECURSIVE select_childs(uuid) AS (
SELECT uuid FROM device_group_folders WHERE uuid = '000B:653D1D5D:00000003'
UNION ALL
SELECT device_group_folders.uuid FROM device_group_folders INNER JOIN select_childs ON parent = select_childs.uuid
) SELECT device_groups.uuid FROM select_childs, device_groups WHERE device_groups.parent = select_childs.uuid;
IN <1 ms
SELECT device_groups.uuid FROM device_groups WHERE device_groups.parent IN (WITH RECURSIVE select_childs(uuid) AS (
SELECT uuid FROM device_group_folders WHERE uuid = '000B:653D1D5D:00000003'
UNION ALL
SELECT device_group_folders.uuid FROM device_group_folders INNER JOIN select_childs ON parent = select_childs.uuid
) SELECT * FROM select_childs);
I have 3 tables :
table_product (30 000 row)
---------
ID
label
_
table_period (225 000 row)
---------
ID
date_start
date_end
default_price
FK_ID_product
and
table_special_offer (10 000 row)
-----
ID
label
date_start,
date_end,
special_offer_price
FK_ID_period
So I need to load data from all these table, so here it's what I do :
1/ load data from "table_product" like this
select *
from table_product
where label like 'gun%'
2/ load data from "table_period" like this
select *
from table_period
where FK_ID_product IN(list of all the ids selected in the 1)
3/ load data from "table_special_offer" like this
select *
from table_special_offer
where FK_ID_period IN(list of all the ids selected in the 2)
As you may think the IN clause in the point 3 can be very very big (like 75 000 big), so I got a lot of chance of getting either a timeout or something like " An expression services limit has been reached".
Have you ever had something like this, and how did you manage to avoid it ?
PS :
the context : SQL server 2005, .net 2.0
(please don't tell me my design is bad, or I shouldn't do "select *", I just simplified my problem so it is a little bit simpler than 500 pages describing my business).
Thanks.
Switch to using joins:
SELECT <FieldList>
FROM Table_Product prod
JOIN Table_Period per ON prod.Id = per.FK_ID_Product
JOIN Table_Special_Offer spec ON per.ID = spec.FK_ID_Period
WHERE prod.label LIKE 'gun%'
Something you should be aware of is the difference of IN vs JOIN vs EXISTS - great article here.
In finally have my answer : table variable (a bit like #smirkingman's solution but not with cte) so:
declare #product(id int primary key,label nvarchar(max))
declare #period(id int primary key,date_start datetime,date_end datetime,defaultprice real)
declare #special_offer(id int,date_start datetime,date_end datetime,special_offer_price real)
insert into #product
select *
from table_product
where label like 'gun%'
insert into #period
select *
from table_period
where exists(
select * from #product p where p.id = table_period.FK_id_product
)
insert into #special_offer
select *
from table_special_offer
where exists(
select * from #period p where p.id = table_special_offer.fk_id_period
)
select * from #product
select * from #period
select * from #special_offer
this is for the sql, and with c# I use ExecuteReader, Read, and NextResult of the class sqldatareader
http://msdn.microsoft.com/en-us/library/system.data.sqlclient.sqldatareader.aspx
I got all I want :
- my datas
- i don't have too much data (unlike the solutions with join)
- i don't execute twice the same query (like solution with subquery)
- i don't have to change my mapping code (1row = 1 business object)
Don't use explicit list of values in IN clause. Instead, write your query like
... FK_ID_product IN (select ID
from table_product
where label like 'gun%')
SELECT *
FROM
table_product tp
INNER JOIN table_period tper
ON tp.ID = tper.FK_ID_product
INNER JOIN table_special_offer so
ON tper.ID = so.FK_ID_period
WHERE
tp.label like 'gun%'"
First some code...
Using JOIN:
SELECT
table_product.* --'Explicit table calls just for organisation sake'
, table_period.*
, table_special_offer.*
FROM
table_product
INNER JOIN table_period
ON table_product.ID = table_period.FK_ID_product
INNER JOIN table_special_offer
ON table_period.ID = table_special_offer.FK_ID_period
WHERE
tp.label like 'gun%'"
Using IN :
SELECT
*
FROM
table_special_offer
WHERE FK_ID_period IN
(
SELECT
FK_ID_period
FROM
table_period
WHERE FK_ID_product IN
(
SELECT
FK_ID_product
FROM
table_product
WHERE label like '%gun'
) AS ProductSub
) AS PeriodSub
Depending on how well your tables get indexed both can be used. Inner Joins as the others have suggested are definitely efficient at doing your query and returning all data for the 3 tables. If you are only needing To use the ID's from table_product and table_period Then using the nested "IN" statements can be good for adapting search criteria on indexed tables (Using IN can be ok if the criteria used are integers like I assume your FK_ID_product is).
An important thing to remember is every database and relational table setup is going to act differently, you wont have the same optimised results in one db to another. Try ALL the possibilities at hand and use the one that is best for you. The query analyser can be incredibly useful in times like these when you need to check performance.
I had this situation when we were trying to join up customer accounts to their appropriate addresses via an ID join and a linked table based condition (we had another table which showed customers with certain equipment which we had to do a string search on.) Strangely enough it was quicker for us to use both methods in the one query:
--The query with the WHERE Desc LIKE '%Equipment%' was "joined" to the client table using the IN clause and then this was joined onto the addresses table:
SELECT
Address.*
, Customers_Filtered.*
FROM
Address AS Address
INNER JOIN
(SELECT Customers.* FROM Customers WHERE ID IN (SELECT CustomerID FROM Equipment WHERE Desc LIKE '%Equipment search here%') AS Equipment ) AS Customers_Filtered
ON Address.CustomerID = Customers_Filtered.ID
This style of query (I apologise if my syntax isn't exactly correct) ended up being more efficient and easier to organise after the overall query got more complicated.
Hope this has helped - Follow #AdaTheDev 's article link, definitely a good resource.
A JOIN gives you the same results.
SELECT so.Col1
, so.Col2
FROM table_product pt
INNER JOIN table_period pd ON pd.FK_ID_product = pt.ID_product
INNER JOIN table_special_offer so ON so.FK_ID_Period = pd.ID_Period
WHERE pt.lable LIKE 'gun%'
I'd be interested to know if this might make an improvement:
WITH products(prdid) AS (
SELECT
ID
FROM
table_product
WHERE
label like 'gun%'
),
periods(perid) AS (
SELECT
ID
FROM
table_period
INNER JOIN products
ON id = prdid
),
offers(offid) AS (
SELECT
ID
FROM
table_special_offer
INNER JOIN periods
ON id = perid
)
... just a suggestion...