I'm looking for a solution where a query should return:
a) a limited set of rows when there are rows in the lowest joined table
b) all rows if there is no data in the lowest joined table
c) taking into account that it is possible that there is more than 1 such join
Objective:
we are implementing security using data. Rows from the table (MainTable) are filtered on 1 or more columns. These columns have a relationship with other tables (LookupTable). Security is defined on the LookupTable.
Example1: the MainTable contains contact information. One of the columns holds the country code, this column has a relationship with a LookupTable that contains the country codes. The user can only select a country code that exists in the LookupTable. The security admin can then define that a user can only work with contacts of one or more countries. When that user accesses the MainTable he/she will only get the contacts of that limited list of countries.
Example2: the MainTable contains products. One column holds the country of origin code, another column the product group. Security setup can limit the access to the product MainTable of a user to a list of countries AND a list of product groups.
The security setup works by Management-by-Exception, whichs means that the MainTable is filtered when one or more "security filters" are defined but if no security filters are defined then the user will get ALL rows from MainTable. So my query should return a limited number of rows if any security filter is defined but should return all rows if there are no security filters defined.
Current situation:
I have been working on a query for the case of Example2. There are 4 possible scenarios:
No security filters are defined
expected outcome: all rows are returned
Security filter defined only for first LookupTable
expected outcome: only rows matching values between LookupTable1 and security filter are returned
Security filter defined only for second LookupTable
expected outcome: only rows matching values between LookupTable2 and security filter are returned
Security filter defined only for both LookupTables
expected outcome: only rows matching values between LookupTable1 AND LookupTable2 and security filter are returned
The query I have is correct for cases 2,3 and 4 but fails for case 1 where no rows are returned (as per my understanding this is due to the fact that both JOINS return an empty result set).
Background:
The application provides the (power) users with some kind of table designer which means that they can define which columns are linked to a LookupTable and which of these LookupTables can be used for the "security filters".
This means that, potentially, we could have a MainTable with for example 200 columns of which 20 are linked to a LookupTable which are defined as security filter. The queries are stored procedures which are generated when "design" changes are saved.
With the query I have now (working for 3 out 4 cases) the number of scenarios is equal to 2^N where N is the number of LookupTables. If N is 20 the total goes over 1 million.
Security setup is done with Profiles assigned to Users and Filter Sets assigned to Profiles and Filter Set Entries containing the actual values to filter on (if any).
The environment is currently on MS SQL 2017 but will be put into production on SQL on Azure.
Example of the query (but look further below for the link to dbfiddle):
SELECT E.col_pk, E.col_28, E.col_7, E.col_8, E.col_9, E.col_1052
FROM MainTable AS E
LEFT JOIN LookupTable2 AS L28 ON L28.col_pk = E.col_28
JOIN SecUserProfile AS UP28 ON UP28.IdentityUserId = #UserId
JOIN SecProfileFilterSets AS PFS28 ON PFS28.SecProfileId = UP28.SecProfileId
LEFT JOIN SecFilterSetEntry AS SE28 ON SE28.SecFilterSetId = PFS28.SecFilterSetId AND SE28.MdxEntityId = 2 AND SE28.EntityKey = L28.col_pk
LEFT JOIN LookupTable13 AS L1052 ON L1052.col_pk = E.col_1052
JOIN SecUserProfile AS UP1052 ON UP1052.IdentityUserId = #UserId
JOIN SecProfileFilterSets AS PFS1052 ON PFS1052.SecProfileId = UP1052.SecProfileId
LEFT JOIN SecFilterSetEntry AS SE1052 ON SE1052.SecFilterSetId = PFS1052.SecFilterSetId AND SE1052.MdxEntityId = 13 AND SE1052.EntityKey = L1052.col_pk
WHERE
(SE28.SecFilterSetId IS NOT NULL AND SE1052.SecFilterSetId IS NOT NULL)
OR
(
SE28.SecFilterSetId IS NOT NULL AND
NOT EXISTS
(
SELECT TOP 1 NUP1052.Id FROM SecUserProfile AS NUP1052
JOIN SecProfileFilterSets AS NPFS1052 ON NPFS1052.SecProfileId = NUP1052.SecProfileId
JOIN SecFilterSetEntry AS NSE1052 ON NSE1052.SecFilterSetId = NPFS1052.SecFilterSetId AND NSE1052.MdxEntityId = 13
WHERE NUP1052.IdentityUserId = #UserId
)
)
OR
(
NOT EXISTS
(
SELECT TOP 1 NUP28.Id FROM SecUserProfile AS NUP28
JOIN SecProfileFilterSets AS NPFS28 ON NPFS28.SecProfileId = NUP28.SecProfileId
JOIN SecFilterSetEntry AS NSE28 ON NSE28.SecFilterSetId = NPFS28.SecFilterSetId AND NSE28.MdxEntityId = 2
WHERE NUP28.IdentityUserId = #UserId
)
AND SE1052.SecFilterSetId IS NOT NULL
)
OR
(
NOT EXISTS
(
SELECT TOP 1 NUP28.Id FROM SecUserProfile AS NUP28
JOIN SecProfileFilterSets AS NPFS28 ON NPFS28.SecProfileId = NUP28.SecProfileId
JOIN SecFilterSetEntry AS NSE28 ON NSE28.SecFilterSetId = NPFS28.SecFilterSetId AND NSE28.MdxEntityId = 2
WHERE NUP28.IdentityUserId = #UserId
)
AND
NOT EXISTS
(
SELECT TOP 1 NUP1052.Id FROM SecUserProfile AS NUP1052
JOIN SecProfileFilterSets AS NPFS1052 ON NPFS1052.SecProfileId = NUP1052.SecProfileId
JOIN SecFilterSetEntry AS NSE1052 ON NSE1052.SecFilterSetId = NPFS1052.SecFilterSetId AND NSE1052.MdxEntityId = 13
WHERE NUP1052.IdentityUserId = #UserId
)
)
Issue:
I have the following issues but they probably boil down to 1 in the end:
my current query is only 75% correct
even if my current query is correct it cannot be used in production with the potential high(er) number of lookup tables.
performance needs to be taken into account. Just as we don't know the number of columns and lookup tables at design time we don't know how many rows the tables will contain. The main table may hold 500, 50000 or 500000 records.
In the end all this will boil down to the right solution :)
I think this is not the easiest of questions (otherwise I will feel very stupid) and for those willing to take a look I've prepared a sandbox environment on dbfiddle representing the use-case I'm working with. I've setup the query to run 4 times, once for each of the scenarios.
Related
I am trying to do a query on three different tables.
The variable table
The variable table carries information about what "area", "rounds" and
"days" the variable belongs to. The variable table also holds a pk column.
The pk is used to determine which variable a record belongs to.
The area table
The area table carries information about the "name" of the area as well as
the "role" the area belongs to. A user is assigned a role and then has
access to specific areas.
The record table
The record table carries information about the record that was entered. It
contains the "value", "alarmed", and "alarmType" columns. You can search
for a record based on the variable, round and day.
I am trying to query all of the variables in a certain round and day for a user.
I want to display all the variables whether or not there is a record found. Currently I have a query that only returns the variables that have records, but not the ones that don't.
If there is no record then thevalue, alarmed, and alarmType column should be null.
This is the query that I have so far constructed:
SELECT DISTINCT variable.name, area.name AS "areaName", variable.pk, CAST(record.value AS TEXT) AS "value", record.alarmed, record.alarmType
FROM variable, area, record
WHERE variable.round LIKE '%,1,%'
AND variable.day LIKE '%,3,%'
AND variable.readOnly = 0
AND variable.area IN (SELECT pk
FROM area
WHERE role = (SELECT role
FROM user
WHERE userName LIKE 'Leo'))
AND variable.area = area.pk
AND record.value = (SELECT CASE WHEN COUNT() < 1 THEN NULL
ELSE CAST(value AS TEXT) END
FROM record
WHERE round = 1
AND day = "11-14-2018"
AND variable = variable.pk)
ORDER BY variable.area, variable.position ASC;
Currently it returns something like this:
There are a lot more variables and I want to know how to display them even if there are no records.
I think I see what you're trying to do. The key is using joins (specifically OUTER joins) instead of trying to mash all the tables together and then find similarities. There are also LEFT, RIGHT and INNER flavors (read more about these here and here), depending on what you consider the "complete" or "master" data set - the starting point of your query.
Here's how I understand your relationships (let me know if I have this wrong):
record.variable --> variable.pk
variable.area --> area.pk
area.role --> user.role
In your case, you stated that you need all records from the variable table, so I would start with this:
SELECT v.*
FROM variable v;
Then, you might find all the AREA records related to a particular USER. Use an INNER join to find only records that exist on BOTH sides of the join:
SELECT a.*, u.*
FROM area a
INNER JOIN user u -- Define the table to join
ON a.role = u.role -- Which columns contain keys to match on
WHERE u.userName = 'Leo';
The WHERE filter applies to the user table, but because we are ONLY asking for records from the area table that have a match with user, then that limits the results from the area table.
The next step is to join these two extracts together using another INNER join, again, to find the intersection - matches that exist on BOTH sides of the join(s):
SELECT v.*, a.*, u.*
FROM variable v -- New starting point
INNER JOIN area a
ON a.pk = v.area
INNER JOIN user u
ON a.role = u.role
WHERE u.userName = 'Leo';
Now, we find all the records for a certain day by adding WHERE clauses:
SELECT v.*, a.*, u.*
FROM variable v
INNER JOIN area a
ON a.pk = v.area
INNER JOIN user u
ON a.role = u.role
WHERE u.userName = 'Leo'
AND v.round = 1 -- Add filters for "round"
AND v.day = '11-14-2018'; -- and "day" columns
Next, we use a LEFT join to give us all the records from the table on the "left" plus any matches we find on the "right" side (the "record" table) or NULL if no match is made:
SELECT v.name
,a.name as "areaName"
,CAST(r.value as TEXT) as "value"
,r.alarmed
,r.alarmType
FROM variable v
INNER JOIN area a
ON v.area = a.pk
INNER JOIN user u
ON a.role = u.role
LEFT JOIN record r -- LEFT is important here
ON v.pk = r.variable
WHERE u.userName = 'Leo'
AND v.round = 1
AND v.day = '11-14-2018'
ORDER BY v.area, v.position;
The result from INNER joins (variable + area + user) becomes the "left" side of this join, and the record becomes the "right" side. Using the LEFT join declares that we want ALL records from the left, whether they have a match on the right or not.
I don't have a dataset to test this with, so please excuse any errors I've made.
Hopefully, this illustrates how joins would be used to both eliminate rows and add data (columns) the result, without having to make individual queries or resort to sub-queries (using IN or EXISTS).
If I have table 1 called "FILES" and table 2 called "Networks" and the value that goes in FILES.Network_ID must be from a list defined in Networks.Network_ID
If I wanted to run a query to find out if there are any values under network_id in table 1 that are not defined in table 2, I tried the below which doesn't work apparently...Also, I'm using the SQL view of MS ACCESS
SELECT *
FROM (FILES f LEFT JOIN Networks
ON f.Network_ID <> Networks.Network_ID)
This query won't work properly because your are trying to join if the ids don't match, so ideally while making join the rows will match
For instance if table Files contains rows with network id 1,2,3 and table Networks contains network id 1,2 then upon joining you will get a combination of rows with network ids [1,2], [2,1],[3,1],[3,2].
Instance if you use a subquery and then filter it using IN clause will work
select * from files where network_id not in (select network_id from networks);
Try this
SELECT *
FROM (FILES f LEFT JOIN Networks
ON f.Network_ID =Networks.Network_ID)
Where Networks.Network_ID Is Null
To ensure data integrity create a relationship between both tables.
I have two tables, with about 20 columns each
users:
id_user user ..... status token
----------------------------------
2 A 0 XdAQ
posts:
id_user post ..... status token
-------------------------------------
3 hi 1 sDyTMl
Query:
SELECT u.*,p.*
FROM posts as p
LEFT JOIN users as u ON u.id_user = p.id_user
WHERE p.id_post = 3
LIMIT 1
So in Php, it could be retrieved any value
....
$status=$a['status'];
$token=$a['token'];
I want to return all the fields of each table to make the post content, the problem is that there is conflict among those identical column names in each table. there are more than 20 columns in each in my real tables, so writing the column names with aliases I think is not the way to go. Is there a way to alias only those identical columns in conflict?
You really should list the specific columns that you want. This is the safest way to retrieve values from the table.
If the only column that is in common is the one used for the join, you can use the USING clause:
SELECT *
FROM posts p LEFT JOIN
users as u
USING (id_user)
WHERE p.id_post = 3
LIMIT 1;
The USING clause is ANSI standard, but not all databases support it. When you use it, only one version of id_post is in the columns returned by the SELECT *. In a LEFT JOIN, it is the version with a value.
If you have other columns with the same name, you need to use column aliases. One short-cut is to take all columns from one table and name the columns in the other:
SELECT u.*, p.col1 as p_col1, . . .
FROM posts p LEFT JOIN
users as u
USING (id_user)
WHERE p.id_post = 3
LIMIT 1;
I have 2 tables. One has been pruned to show only ID's which meet certain criteria. The second needs to be pruned to show only data that matches the previous "array" of id's. there can be multiple results.
Consider the following:
Query_1_final: Returns the ID's of users whom meet certain criteria:
select
t1.[user_id]
from
[SQLDB].[db].[meeting_parties] as t1
inner join
(select distinct
[user_id]
from
[SQLDB].[db].[meeting_parties]
group by
[user_id]
having
count([user_id]) = 1) as t2 on t1.user_id = t2.user_id
where
[type] = 'organiser'
This works great and returns:
user_id
--------------------
22
1255
9821
and so on...
It produces a single column with the ID's of everyone who is a "Meeting Organizer" and also in the active_meetings table. (note, there are multiple types/roles, this was the best way to grab them all)
Now, I need this data to filter another table, another join. Here is the start of my query
Query_2_PREP: returns 5 columns where the meeting has "started" already.
SELECT
[meeting_id]
,[meeting_style]
,[meeting_day]
,[address]
,[promos]
FROM
[SQLDB].[db].[all_meetings]
WHERE
[meeting_started] = 'TRUE'
This works as well
meeting_id | meeting_style | meeting_day ...
---------------------------------------------
23 open M,F,SA
23 discussion TU,TH
23 lead W,F
and so on...
and returns ALL 10,982 meetings that started, but I need it to return only the meetings that are from the distinct 'organiser's ID's from Query_1_final (which should be more like 1200 records or so)
Ideally, I need something "like" this below (but of course it does not work)
Query 2: needs to return all meetings that are from organiser ID's only.
SELECT
[meeting_party_id]
,[meeting_style]
,[meeting_day]
,[address]
,[promos]
FROM
[SQLDB].[db].[all_meetings]
WHERE
[meeting_started] = 'TRUE'
AND [meeting_party_id] = "ANY Query_1_final results, especially multiple"
I have tried nesting JOIN and INNER JOIN's but I think there is something fundamental I am missing here about SQL. In PHP I would use an array compare or just run another query... any help would be much appreciated.
Just use IN. Here is the structure of the logic:
with q1 as (
<first query here>
)
SELECT m.*
FROM [SQLDB].[db].[all_meetings] m
WHERE meeting_started = 'TRUE' AND
meeting_party_id IN (SELECT user_id FROM q1);
I have 3 tables:
tbl_Image from which a list of all images will be obtained
A user table from which User ID will be obtained
and an association table of Image and Member called tbl_MemberAssociation.
My work flow is that a user can upload image and this will be stored in to image table. Then all users can view this image and select one of three choice provided along with the image. If user selects an option it will be added to Association table. No user can watch same image more than once. So multiple entries will not be there.
Now I want to find the % of match by getting the list of members choose the same option and different option corresponding to all common images for which they have provided their option.
I.e. say if 3 users say A, B and C view an image of tajmahal. If A and B opted beautiful as choice and C as "Not Good ". For another image say Indian Flag A B and C opted same as Salute. Then for User A: B have 100 % match (since they selected same option both times). For A : C have 50% match one among 2 same.
So this is my scenario, in which I have to find all matched corresponding to currently logged in User.
Please help me.... I am totally disturbed with this procedure.
I have made some assumptions about the actual structure of your tables, but if I understand what you are looking for then I think this query will get the results you are wanting. You may have to make a few modifications to match your table structures.
SELECT
matches.UserName,
CAST(matches.SameRatings AS FLOAT) / CAST(ratings.UserRatingCount AS FLOAT) AS MatchPercent
FROM
tbl_User
CROSS APPLY
(
SELECT
COUNT(*) UserRatingCount
FROM
tbl_MemberAssociation
WHERE
UserId = tbl_User.UserId
) ratings
CROSS APPLY
(
SELECT
u1.UserId,
u1.UserName,
COUNT(*) AS SameRatings
FROM
tbl_MemberAssociation ma
INNER JOIN
tbl_MemberAssociation ma1
ON
ma.ImageId = ma1.ImageId
AND ma.Rating = ma1.Rating
AND ma.UserId <> ma1.UserId
INNER JOIN
tbl_User u1
ON
ma1.userId = u1.UserId
WHERE
ma.UserId = tbl_User.UserId
GROUP BY
u1.UserId,
u1.UserName
) matches
WHERE
tbl_User.UserId = #UserId
ORDER BY
MatchPercent DESC
#UserId could be passed as an input to the stored procedure.
The 1st CROSS APPLY "ratings" is getting a count of for the total number of ratings for the logged in user.
The 2nd CROSS APPLY "matches" is getting a count of the number of like ratings for the other users in the database.
The result set uses the counts calculated by the two CROSS APPLY queries to compute the match percentage between the logged in user and the other users who have rated the same images as the logged in user.