An Exclude Query with Link tables - sql

I have two tables
**Item**
ID
...
**ShapeItem**
ID
ItemID
ShapeID
RegionID
(There are other columns and tables but they are not relevant to this question)
I can return all items that have a specific shape id in a specific region id successfully using an INNER JOIN
SELECT ID FROM Item INNER JOIN ShapeItem ON Item.ID = ShapeItem.ItemID WHERE ShapeID = 2
However I want to reverse this logic and return all items that DONT have a specific shape so my first thought was to do
SELECT ID FROM Item INNER JOIN ShapeItem ON Item.ID = ShapeItem.ItemID WHERE ShapeID <> 2
however this did not produce the required result, this returned all items that had a shape that was not the specific shape, but it did not account for those items that did not have any shapes at all.
My next thought was to use a LEFT JOIN but this returned every item with null values, (over 400,000)
I am currently stuck on this, can you suggest a way forward for me please?
Summary
I want to return all items that dont have a specific shape, including those items that are not referenced at all in the ShapeItem table.
SQL SERVER COMPACT 4.0
C#
Visual Studio 2012

Related

Joining three tables using result from one SQL query on one of the tables

I'm trying to search one table for entities with a certain field in one table and use fields within that entity to to join it with two other tables specific fields
first I search for entities with CustomerID == 2 and use their other fields to search to other tables
Using BoxTypeID from those specific entities I want to find the corresponding BoxType
Using WarehouseID from those specific entities I want to find the corresponding name
Once I have found all three fields (BoxID, BoxType, name) I want to display them all in one table
I have some SQL that works but only displays the BoxID and Warehouses.name. I can get it to also display BoxID and BoxType however not all three at once.
SELECT Boxes.BoxID, Warehouses.name AS Warehouse
FROM Boxes
INNER JOIN Warehouses
ON Boxes.WarehouseID = Warehouses.WarehouseID
WHERE Boxes.CustomerID = 2;
resulting table from query above
However once I try to apply another JOIN it doesn't work
SELECT Boxes.BoxID, Warehouses.name AS Warehouse, BoxTypes.BoxType as Size
FROM Boxes
INNER JOIN Warehouses
ON Boxes.WarehouseID = Warehouses.WarehouseID
INNER JOIN BoxTypes
On Boxes.BoxTypeID = BoxTypes.BoxTypeID
Where Boxes.CustomerID = 2;

SELECT query with conditional joins, return all rows when no data in lowest join

I'm looking for a solution where a query should return:
a) a limited set of rows when there are rows in the lowest joined table
b) all rows if there is no data in the lowest joined table
c) taking into account that it is possible that there is more than 1 such join
Objective:
we are implementing security using data. Rows from the table (MainTable) are filtered on 1 or more columns. These columns have a relationship with other tables (LookupTable). Security is defined on the LookupTable.
Example1: the MainTable contains contact information. One of the columns holds the country code, this column has a relationship with a LookupTable that contains the country codes. The user can only select a country code that exists in the LookupTable. The security admin can then define that a user can only work with contacts of one or more countries. When that user accesses the MainTable he/she will only get the contacts of that limited list of countries.
Example2: the MainTable contains products. One column holds the country of origin code, another column the product group. Security setup can limit the access to the product MainTable of a user to a list of countries AND a list of product groups.
The security setup works by Management-by-Exception, whichs means that the MainTable is filtered when one or more "security filters" are defined but if no security filters are defined then the user will get ALL rows from MainTable. So my query should return a limited number of rows if any security filter is defined but should return all rows if there are no security filters defined.
Current situation:
I have been working on a query for the case of Example2. There are 4 possible scenarios:
No security filters are defined
expected outcome: all rows are returned
Security filter defined only for first LookupTable
expected outcome: only rows matching values between LookupTable1 and security filter are returned
Security filter defined only for second LookupTable
expected outcome: only rows matching values between LookupTable2 and security filter are returned
Security filter defined only for both LookupTables
expected outcome: only rows matching values between LookupTable1 AND LookupTable2 and security filter are returned
The query I have is correct for cases 2,3 and 4 but fails for case 1 where no rows are returned (as per my understanding this is due to the fact that both JOINS return an empty result set).
Background:
The application provides the (power) users with some kind of table designer which means that they can define which columns are linked to a LookupTable and which of these LookupTables can be used for the "security filters".
This means that, potentially, we could have a MainTable with for example 200 columns of which 20 are linked to a LookupTable which are defined as security filter. The queries are stored procedures which are generated when "design" changes are saved.
With the query I have now (working for 3 out 4 cases) the number of scenarios is equal to 2^N where N is the number of LookupTables. If N is 20 the total goes over 1 million.
Security setup is done with Profiles assigned to Users and Filter Sets assigned to Profiles and Filter Set Entries containing the actual values to filter on (if any).
The environment is currently on MS SQL 2017 but will be put into production on SQL on Azure.
Example of the query (but look further below for the link to dbfiddle):
SELECT E.col_pk, E.col_28, E.col_7, E.col_8, E.col_9, E.col_1052
FROM MainTable AS E
LEFT JOIN LookupTable2 AS L28 ON L28.col_pk = E.col_28
JOIN SecUserProfile AS UP28 ON UP28.IdentityUserId = #UserId
JOIN SecProfileFilterSets AS PFS28 ON PFS28.SecProfileId = UP28.SecProfileId
LEFT JOIN SecFilterSetEntry AS SE28 ON SE28.SecFilterSetId = PFS28.SecFilterSetId AND SE28.MdxEntityId = 2 AND SE28.EntityKey = L28.col_pk
LEFT JOIN LookupTable13 AS L1052 ON L1052.col_pk = E.col_1052
JOIN SecUserProfile AS UP1052 ON UP1052.IdentityUserId = #UserId
JOIN SecProfileFilterSets AS PFS1052 ON PFS1052.SecProfileId = UP1052.SecProfileId
LEFT JOIN SecFilterSetEntry AS SE1052 ON SE1052.SecFilterSetId = PFS1052.SecFilterSetId AND SE1052.MdxEntityId = 13 AND SE1052.EntityKey = L1052.col_pk
WHERE
(SE28.SecFilterSetId IS NOT NULL AND SE1052.SecFilterSetId IS NOT NULL)
OR
(
SE28.SecFilterSetId IS NOT NULL AND
NOT EXISTS
(
SELECT TOP 1 NUP1052.Id FROM SecUserProfile AS NUP1052
JOIN SecProfileFilterSets AS NPFS1052 ON NPFS1052.SecProfileId = NUP1052.SecProfileId
JOIN SecFilterSetEntry AS NSE1052 ON NSE1052.SecFilterSetId = NPFS1052.SecFilterSetId AND NSE1052.MdxEntityId = 13
WHERE NUP1052.IdentityUserId = #UserId
)
)
OR
(
NOT EXISTS
(
SELECT TOP 1 NUP28.Id FROM SecUserProfile AS NUP28
JOIN SecProfileFilterSets AS NPFS28 ON NPFS28.SecProfileId = NUP28.SecProfileId
JOIN SecFilterSetEntry AS NSE28 ON NSE28.SecFilterSetId = NPFS28.SecFilterSetId AND NSE28.MdxEntityId = 2
WHERE NUP28.IdentityUserId = #UserId
)
AND SE1052.SecFilterSetId IS NOT NULL
)
OR
(
NOT EXISTS
(
SELECT TOP 1 NUP28.Id FROM SecUserProfile AS NUP28
JOIN SecProfileFilterSets AS NPFS28 ON NPFS28.SecProfileId = NUP28.SecProfileId
JOIN SecFilterSetEntry AS NSE28 ON NSE28.SecFilterSetId = NPFS28.SecFilterSetId AND NSE28.MdxEntityId = 2
WHERE NUP28.IdentityUserId = #UserId
)
AND
NOT EXISTS
(
SELECT TOP 1 NUP1052.Id FROM SecUserProfile AS NUP1052
JOIN SecProfileFilterSets AS NPFS1052 ON NPFS1052.SecProfileId = NUP1052.SecProfileId
JOIN SecFilterSetEntry AS NSE1052 ON NSE1052.SecFilterSetId = NPFS1052.SecFilterSetId AND NSE1052.MdxEntityId = 13
WHERE NUP1052.IdentityUserId = #UserId
)
)
Issue:
I have the following issues but they probably boil down to 1 in the end:
my current query is only 75% correct
even if my current query is correct it cannot be used in production with the potential high(er) number of lookup tables.
performance needs to be taken into account. Just as we don't know the number of columns and lookup tables at design time we don't know how many rows the tables will contain. The main table may hold 500, 50000 or 500000 records.
In the end all this will boil down to the right solution :)
I think this is not the easiest of questions (otherwise I will feel very stupid) and for those willing to take a look I've prepared a sandbox environment on dbfiddle representing the use-case I'm working with. I've setup the query to run 4 times, once for each of the scenarios.

I want to use CONCAT and exclude and duplicate any duplicate entries

I'm trying to use the CONCAT expression, but also exclude any duplicate entries.
So I'm trying to update a report based on a single process held in our product. The problem is that whoever created the tables that the current report is pulling from is not from a single table. Currently I have found three tables that the report pulls from for one column.
SELECT concat(dbo.t_log_TaskBody.TaskDescription,' ', dbo.t_ezDocument.FileName) as Title
FROM dbo.t_logs_SigDocPrintedEmailed
LEFT JOIN dbo.t_log_Data ON t_logs_SigDocPrintedEmailed.t_ezDataPKid = dbo.t_log_Data.PKid
LEFT JOIN dbo.t_log_TaskBody ON dbo.t_logs_SigDocPrintedEmailed.t_ezSignDocumentQ_PKid = dbo.t_log_TaskBody.DocumentId
LEFT JOIN dbo.t_ezSignDocumentQ ON dbo.t_logs_SigDocPrintedEmailed.t_ezSignDocumentQ_PKid = dbo.t_ezSignDocumentQ.PKid
LEFT JOIN dbo.t_ezArcSigDocQLog ON dbo.t_logs_SigDocPrintedEmailed.t_ezSignDocumentQ_PKid = dbo.t_ezArcSigDocQLog.t_ezSignDocQPKID
LEFT JOIN dbo.t_ezDocument ON dbo.t_ezSignDocumentQ.t_ezDocument = dbo.t_ezDocument.PKID or dbo.t_ezArcSigDocQLog.t_ezDocument = EasyID.dbo.t_ezDocument.PKID
So now that I have one entry that happens to connect to two of the tables I'm pulling from to get a title of a document I end up with the title appearing twice in one box. Is there anyway I can use CONCAT to combine the two tables while keeping it to unique entries, or is there a better way of doing this.
I'll get Something along the lines of:
Title
null
Title Title
null
Title
Title
Based on your sample results, I think you want COALESCE(), not CONCAT():
concat(dbo.t_log_TaskBody.TaskDescription, dbo.t_ezDocument.FileName) as Title
You have no examples where results are actually concatenated.

SQL query on one to many relationship with multiple filters

I have some experience with SQL but I still couldn't find out how can I do the following query performance efficient.
I have 2 tables - Box and Item. Box has id attribute which is the primary key (and some more), and Item has box_id, type, name. Each table has billions of records, each box has on average 10 items.
I want to query all the boxes that have at least one item with a given type, and at least one item with a given name (could be the same item or different). The query should be paginated with page size of 10.
I used single column indexing on all Item attributes. The following query for that (the first page) takes a very long duration (more than a minute):
SELECT Box.id FROM Box WHERE
(EXISTS (SELECT 1 FROM Item WHERE Item.box_id = Box.id AND Item.type = 'my_type')) AND
(EXISTS (SELECT 1 FROM Item WHERE Item.box_id = Box.id AND Item.name = 'my_name'))
LIMIT 10
I think that the problem is making the intersection between boxes filtered in each part of the query (querying with just one of the constraints returns about million records). I am using Aurora PostgreSQL 9.6.6.
You haven't responded to the clarifications so I will assume a few things:
You want ALL the boxes, not just 10 of them.
There's a typo when comparing by name. Should be: Item.name = 'my_name'
You said "I have indexed all Item attributes." I would assume you have single column indexes for all the columns of the Item table.
The column id of Box is the primary key, and therefore it already has an index on it.
Now, my take is the indexes you are using are not optimal for this query since they only include columns separately. If you don't already have them, please try creating the following indexes:
create index ix1 on Item (box_id, type);
create index ix2 on Item (box_id, name);
Yes, both of them. Try the query again and see how long does it take.
If still slow, please post the explain plan, using:
EXPLAIN ANALYZE
SELECT Box.id
FROM Box
WHERE
(EXISTS (SELECT 1 FROM Item WHERE Item.box_id = Box.id AND Item.type = 'my_type'))
AND
(EXISTS (SELECT 1 FROM Item WHERE Item.box_id = Box.id AND Item.name = 'my_name'))
INTERSECT is another option.
SELECT Box_id FROM Item
WHERE Item.type = 'my_type'
INTERSECT
SELECT Box_id FROM Item
WHERE Item.name = 'my_name'
Note: INTERSECT returns distinct values so no need for an outer query to get the list of distinct Box_id values that meet your criteria. This query does return orphan items (items with a box_id not in the box table) so an outer query might be required if this is the case.
Something like this?
SELECT DISTINCT ON (Box.id) Box.*
FROM Box
JOIN Item I1 ON I1.box_id = Box.id AND I1.type = 'my_type'
JOIN Item I2 ON I2.box_id = Box.id AND I2.name = 'my_name'
ORDER BY Box.id;
JOINs filters results by item's type and name.

Join List of words to a table

I have two tables. Table "List" and table "Content". I want to store 6 types of lists. These lists are black,white and grey lists and each of these list contain a few words. Whenever someone notices a new word that should be in one of the lists, then new word should be simply added to the database.
The image below shows you the tables that I use.
I want to refer or join a certain list for instance a list with name: "Blacklist" that has an ListID= 1 with the correct set of blacklistwords, that could have a ContentID=1.
The content is a list of words, but I am clueless as for how I should join the correct list of words(content) to a listID. I don't know how to query this.
The part that is troubeling me is that it is a list of words. So a ContentID =1 has for example the words"Login","Password", "Credential" etc. How do I query it to ListID=1 with the name"BlackList"? And do the same for the other lists?
I think it should look like this.
SELECT ID
FROM List
LEFT JOIN Content
ON LIST.ID = ContenID AND CONTENT.ISDEFAULT = 1
WHERE ListID = 1
This only joins the two ID with each other. How do I join the correct list of words with the correct list? Maybe I am totally missing the point with the query above?
Question: How do I join a set or list of words to a list with a name and ListID?
Once you change this schema, the below query will work
SELECT ListID,ContentID,Words
FROM List
LEFT JOIN Content
ON List.ListID = Content.ListID
WHERE List.ListID = 1
I have considered the schema from the diagrams. Please execute the below query:
SELECT
L.Name AS 'ListName',
C.Words
FROM List L
INNER JOIN Content C ON C.ListID = L.ListID
WHERE
C.Words IN ('Login','Password','Credential')