I'm trying design a system for an API that grants users access to a data table Data based on a permission table Permissions which is related to a group table Group. When a user makes a request for data (from the Data table), my API should only return rows from the Data table based on the values within the columns of the Data table that they have been granted to
view.
By default, a user will have no access to any rows when requesting data through my API. However, I'd like to grant access to Data based on values in columns.
For example If my Data table contains information about news articles and has columns title, news_source, posted_date, and other similar columns
id | title | news_source | posted_date | ...
-----+----------+-----------------------+-------------+------
1 | ... | NYTimes | 2019-12-30 |
2 | ... | BBC | 2019-12-30 |
3 | ... | BBC | 2019-12-30 |
4 | ... | Washington Post | 2019-12-30 |
5 | ... | NYTimes | 2019-12-30 |
6 | ... | NYTimes | 2020-01-01 |
7 | ... | Boston Globe | 2020-01-01 |
In this example, I'd like to grant a group access to get data only from NYTimes, posted after 2020-01-01, etc...
To do this, I've implemented the schema below
+-----+ +--------------+
|Group|<-------|Permission |
+-----+ +--------------+
|name | |group_id |
|... | |column_name |
+-----+ |text_value |
|date_value |
+--------------+
For Group, name is just the name of the group and the ellipse represents some other non-relevant columns. In Permissions, I have the foreign key to Group (group_id), the name of the column in the Data table that I'm accessing (column_name), and the value I'm granting access to (text_value or date_value depending on the column I'm referencing).
Right now, when a user makes a request for data, I run this SQL to apply the permissions (if the user's group has id = 1).
SELECT * FROM Data d
INNER JOIN Permission p1 ON p1.group_id = 1 AND p1.column_name = 'news_source' AND p1.text_value = d.news_source
INNER JOIN Permission p2 ON p2.group_id = 1 AND p2.column_name = 'posted_date' AND p2.date_value >= d.posted_date;
This will work, but I was wondering if there was a better more organized way to go about this. I feel there would be a lot of redundancy in this model across multiple groups with the same permissions.
I'm trying to create a stored procedure that will retrieve a table (called CommsLog) and match it to a user table to return all the names that are associated with it.
The user database stores all the users by alias and then I am trying to look up their first and last name in the database, concatenate them and return that into my results
This is what I've got at the moment but it only returns part of the table and the same names for both columns (These should be different)
SELECT
CommsLog.ID,
CommsLog.CommType,
CommsLog.Date,
Users.FirstName +' ' + Users.LastName,
Users.FirstName +' ' + Users.LastName,
CommsLog.Version
FROM CommsLog
INNER JOIN Users
ON CommsLog.SentTo=Users.Alias and CommsLog.SentFrom=.Users.Alias
EDIT: Update with Data and output
CommsLog table looks like:
+----+----------+--------------------------+----------+----------+---------+
| ID | CommType | Date | SentTo | SentFrom | Version |
+----+----------+--------------------------+----------+----------+---------+
| 12 | Test | 2014-12-19 09:38:10.000 | uk\tmot | uk\gmab | 1.10 |
+----+----------+--------------------------+----------+----------+---------+
User table looks like:
+---------+-----------+----------+------------+----------------------------------+
| Alias | FirstName | LastName | Telephone | email |
+---------+-----------+----------+------------+----------------------------------+
| uk\tmot | Tom | motoll | 0731424523 | tom.motoll#stackoverflow.com |
| uk\gmab | Grant | maberick | 0756463345 | grant.maberick#stackoverflow.com |
+---------+-----------+----------+------------+----------------------------------+
Ouput:
+----+----------+-------------------------+------------+----------------+---------+
| ID | CommType | Date | SentTo | SentFrom | Version |
+----+----------+-------------------------+------------+----------------+---------+
| 12 | Test | 2014-12-19 09:38:10.000 | Tom motoll | Grant maberick | 1.10 |
+----+----------+-------------------------+------------+----------------+---------+
It looks to me like you're trying to pull in both the names of the SentTo users as well as the names of the SentFrom users in your SELECT statement. If that's the case, then you're actually going to need to join your USERS table into your query twice, with aliases -- once for your SentTo users and once for your SentFrom users. Try this for a query, instead:
SELECT
CSLL.CommsLog.ID,
CSLL.CommsLog.CommType,
CSLL.CommsLog.Date,
UserTo.FirstName +' ' + UserTo.LastName,
UserFrom.FirstName +' ' + UserFrom.LastName,
CSLL.CommsLog.Version
FROM CSLL.CommsLog
INNER JOIN CSLL.Users UserTo ON CSLL.CommsLog.SentTo=UserTo.Alias
INNER JOIN CSLL.Users UserFrom ON CSLL.CommsLog.SentFrom=UserFrom.Alias
Since you didn't post your table structure or any sample data, you may need to tweak that query a bit to make it work, but it should at least get you close to what you're after.
Are you meaning to only get records that were sent from a user to themselves?
That is what your join condition is doing.
So I have a table which stores ltree entries along with umask-based permissions for them.
| entry | user | group | mask |
| a | 1 | 1 | 644 |
| a.b | 2 | 1 | 644 |
| a.b.c | 2 | 0 | 600 |
Permissions are inheritable, and permission check is currently done client-side, (no caching - whole tree is retrieved to check permissions for given key).
What can be seen as a better workaround?
Using separate table to keep rights (this way) - fast to query, slow to update? (10'000 keys, 100' users, 20-30 groups to help organize users => expecting ~200*10000 keys)
Keep same structure, cache permissions client-side?
Write a stored procedure to query tree based on provided entry,user,group?
something else?
I consider myself fairly competent in understanding and manipulating C-ish languages; it's not a problem for me to come up with an algorithm and implement it in any C-ish language.
I have tremendous difficulty writing SQL (in my specific case, MySQL) queries. For very simple queries, it isn't a problem, but for complex queries, I become frustrated not knowing where to start. Reading the MySQL documentation is difficult, mainly because the syntax description and explanation isn't organized very well.
For example, the SELECT documentation is all over the map: it starts out with what looks like psuedo-BNF, but then (since the text for aggregate descriptions aren't clickable... like select_expr) it quickly devolves into this frustrating exercise of trying to piece the syntax together yourself by having a number of browser windows open.
Enough whining.
I'd like to know how people, step by step, begin constructing a complex MySQL query. Here is a specific example. I have three tables below. I want to SELECT a set of rows with the following characteristics:
From the userInfo and userProgram tables, I want to select the userName, isApproved, and modifiedTimestamp fields and UNION them into one set. From this set I want to ORDER by modifiedTimestamp taking the MAX(modifiedTimestamp) for every user (i.e. there should be only one row with a unique userName and the timestamp associated with that username should be as high as possible).
From the user table, I want to match the firstName and lastName that is associated with the userName so that it looks something like this:
+-----------+----------+----------+-------------------+
| firstName | lastName | userName | modifiedTimestamp |
+-----------+----------+----------+-------------------+
| JJ | Prof | jjprofUs | 1289914725 |
| User | 2 | user2 | 1289914722 |
| User | 1 | user1 | 1289914716 |
| User | 3 | user3 | 1289914713 |
| User | 4 | user4 | 1289914712 |
| User | 5 | user5 | 1289914711 |
+-----------+----------+----------+-------------------+
The closest I've got is a query that looks like this:
(SELECT firstName, lastName, user.userName, modifiedTimestamp
FROM user, userInfo
WHERE user.userName=userInfo.userName)
UNION
(SELECT firstName, lastName, user.userName, modifiedTimestamp
FROM user, userProgram
WHERE user.userName=userProgram.userName)
ORDER BY modifiedTimestamp DESC;
I feel like I'm pretty close but I don't know where to go from here or even if I'm thinking about this in the right way.
> user
+--------------------+--------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+--------------------+--------------+------+-----+---------+-------+
| userName | char(8) | NO | PRI | NULL | |
| firstName | varchar(255) | NO | | NULL | |
| lastName | varchar(255) | NO | | NULL | |
| email | varchar(255) | NO | UNI | NULL | |
| avatar | varchar(255) | YES | | '' | |
| password | varchar(255) | NO | | NULL | |
| passwordHint | text | YES | | NULL | |
| access | int(11) | NO | | 1 | |
| lastLoginTimestamp | int(11) | NO | | -1 | |
| isActive | tinyint(4) | NO | | 1 | |
+--------------------+--------------+------+-----+---------+-------+
> userInfo
+-------------------+------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-------------------+------------+------+-----+---------+-------+
| userName | char(8) | NO | MUL | NULL | |
| isApproved | tinyint(4) | NO | | 0 | |
| modifiedTimestamp | int(11) | NO | | NULL | |
| field | char(255) | YES | | NULL | |
| value | text | YES | | NULL | |
+-------------------+------------+------+-----+---------+-------+
> userProgram
+-------------------+--------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-------------------+--------------+------+-----+---------+-------+
| userName | char(8) | NO | PRI | NULL | |
| isApproved | tinyint(4) | NO | PRI | 0 | |
| modifiedTimestamp | int(11) | NO | | NULL | |
| name | varchar(255) | YES | | NULL | |
| address1 | varchar(255) | YES | | NULL | |
| address2 | varchar(255) | YES | | NULL | |
| city | varchar(50) | YES | | NULL | |
| state | char(2) | YES | MUL | NULL | |
| zip | char(10) | YES | | NULL | |
| phone | varchar(25) | YES | | NULL | |
| fax | varchar(25) | YES | | NULL | |
| ehsChildren | int(11) | YES | | NULL | |
| hsChildren | int(11) | YES | | NULL | |
| siteCount | int(11) | YES | | NULL | |
| staffCount | int(11) | YES | | NULL | |
| grantee | varchar(255) | YES | | NULL | |
| programType | varchar(255) | YES | | NULL | |
| additional | text | YES | | NULL | |
+-------------------+--------------+------+-----+---------+-------+
For what I understand from your question, you seem to need a correlated query, which would look like this:
(SELECT firstName, lastName, user.userName, modifiedTimestamp
FROM user, userInfo ui1
WHERE user.userName=userInfo.userName
AND modifiedtimestamp=(select max(modifiedtimestamp) from userInfo ui2 where ui1.userName=ui2.userName))
UNION
(SELECT firstName, lastName, user.userName, modifiedTimestamp
FROM user, userProgram up1
WHERE user.userName=userProgram.userName
AND modifiedtimestamp=(select max(modifiedtimestamp) from userProgram up2 where up1.userName=up2.userName))
ORDER BY modifiedTimestamp DESC;
So, do I proceed to get to this result? Key is: express clearly the information you want to retrieve, without taking mental shortcuts.
Step 1: Choose the fields I need in the different tables of my database. That's what is between SELECT and FROM. Seems obvious, but it becomes less obvious when it comes to aggregation function like sums or counts. In that case, you have to say, for example "I need the count of lines in userInfo for each firstName". See below in GROUP BY.
Step 2: Knowing the field you need, write the joins between the different corresponding tables. That's an easy one...
Step 3: Express your conditions. It can be easy, like if you want data from user for userName="RZEZDFGBH", or more complicated, like in your case: the way to formulate it so you can get the thing done, if you want only the most recent modifiedtimestamp, is "so that the modifiedtimestamp is equal to the most recent modifiedtimestamp" (that's where you can easily take a mental shortcut and miss the point)
Step 4: If you have aggregates, it's time to set the GROUP BY statement. For example, if you count all line in userInfo for each firstName, you would write "GROUP BY firstName":
SELECT firstName,count(*) FROM userInfo GROUP BY firstName
This gives you the number of entries in the table for each different firstName.
Step 5: HAVING conditions. These are conditions on the aggregates. In the previous example, if you wanted only the data for the firstName having more than 5 lines in the table, you could write SELECT firstName,count(*) FROM userInfo GROUP BY firstName HAVING count(*)>5
Step 6: Sort with ORDER BY. Pretty easy...
That's only a short summary. There is much, much more to discover, but it would be too long to write an entire SQL course here... Hope it helps, though!
As f00 says, it's simple(r) if you think of the data in terms of sets.
One of the issues with the question as it stands is that the expected output doesn't match the stated requirements - the description mentions the isApproved column, but this doesn't appear anywhere in either the query or the expected output.
What this illustrates is that the first step in writing a query is to have a clear idea of what you want to achieve. The bigger issue with the question as it stands is that this is not clearly described - instead, it moves from a sample table of expected output (which would be more helpful if we had corresponding samples of expected input data) straight into a description of how you intend to achieve it.
As I understand it, what you want to see is a list of users (by username, with their associated first and last names), together with the last time any associated record was modified on either the userInfo or userProgram tables.
(It isn't clear whether you want to see users who have no associated activity on either of these other tables - your supplied query implies not, otherwise the joins would be outer joins.)
So, you want a list of users (by username, with their associated first and last names):
SELECT firstName, lastName, userName
FROM user
together with a list of times that records were last modified:
SELECT userName, MAX(modifiedTimestamp)
...
on either the userInfo or userProgram tables:
...
FROM
(SELECT userName, modifiedTimestamp FROM userInfo
UNION ALL
SELECT userName, modifiedTimestamp FROM userProgram
) subquery -- <- this is an alias
...
by userName:
...
group by userName
These two sets of data need to be linked by their userName - so the final query becomes:
SELECT user.firstName, user.lastName, user.userName,
MAX(subquery.modifiedTimestamp) last_modifiedTimestamp
FROM user
JOIN
(SELECT userName, modifiedTimestamp FROM userInfo
UNION ALL
SELECT userName, modifiedTimestamp FROM userProgram
) subquery
ON user.userName = subquery.userName
GROUP BY user.userName
In most versions of SQL, this query would return an error as user.firstName and user.lastName are not included in the GROUP BY clause, nor are they summarised.
MySQL allows this syntax - in other SQLs, since those fields are functionally dependant on userName, adding a MAX in front of each field or adding them to the grouping would achieve the same result.
A couple of additional points:
UNION and UNION ALL are not identical - the former removes duplicates while the latter does not; this makes the former more processor-intensive.
Since duplicates will be removed by the grouping, it is better to use UNION ALL.
Many people will write this query as user joined to userInfo UNIONed ALL with user joined to userProgram - this is because many SQL engines can optimise this type of query more effectively.
At this point, this represents premature optimisation.
There's a lot of good stuff here. Thanks to everyone who contributed. This is a quick summary of the things I found helpful as well as some additional thoughts in connecting building functions to building queries. I wish I could give everyone SO merit badges/points but I think that there can only be one (answer) so I'm picking Traroth based upon point total and personal helpfulness.
A function can be understood as three parts: input, process, output. A query can be understood similarly. Most queries look something like this:
SELECT stuff FROM data WHERE data is like something
The SELECT portion is the output. There are some capabilities for formatting the output here (i.e. using AS)
The FROM portion is the input. The input should be seen as a pool of data; you will want to make this as specific as possible, using a variety of joins and subqueries that are appropriate.
The WHERE portion is like the process, but there's a lot of overlap with the FROM portion. Both the FROM and WHERE portions can reduce the pool of data appropriately using a variety of conditions to filter out unwanted data (or to only included desired data). The WHERE portion can also help format the output.
Here's how I broke down the steps:
Start with thinking about what your output looks like. This stuff goes into the SELECT portion.
Next, you want to define the set of data that you wish to work on. Traroth notes: "Knowing the field you need, write the joins between the different corresponding tables. That's an easy one..." It depends on what you mean by 'easy'. If you are new to writing queries, you will probably just default to writing inner joins (like I did). This is not always the best way to go. http://en.wikipedia.org/wiki/Join_(SQL) is a great resource to understanding the different kinds of joins possible.
As a part of the previous step think about smaller parts of that data set and build up to the complete data set you are interested in. In writing a function, you can write subfunctions to help express your process in a clearer manner. Similar to that, you can write subqueries. A huge tip from Mark Bannister in creating a subquery AND USING AN ALIAS. You will have to reconfigure your output to use this alias, but this is pretty key.
Last, you can use various methods to pare down your data set, removing data you're not interested in
One way to think about the data you are operating on is a giant 2-D matrix: JOINs make larger the horizontal aspect, UNIONs make larger the vertical aspect. All the other filters are designed to make this matrix smaller to be appropriate for your output. I don't know if there is a "functional" analogy to JOIN, but UNION is just adding the output of two functions together.
I realize, though, there are lots of ways that building query IS NOT like writing a function. For example, you can build and pare down your data set in both the FROM and WHERE areas. What was key for me was understanding joins and finding out how to create subqueries using aliases.
just learn to think in terms of sets - then it's simple :P
http://www.codinghorror.com/blog/2007/10/a-visual-explanation-of-sql-joins.html
You can't construct sql without understanding the data in the tables and the logical result required. There's no background given for what data the tables might look like and mean and the description of the results you're trying to gather doesn't make sense to me so I'm not going to venture a guess.
On the latter point... it's rare that you'd want a union of timestamp values multiple sources. Generally speaking when results like that are gathered it's generally for some sort of auditing/tracing. However, when you're discarding all information about the source of the timestamp and just computing a maximum you have... well what exactly?
Anyways, one or more examples of data and desired output and maybe something about the application and the whys is a must to make yourself clear.
To the extent I'll make any prediction about the shape of your eventual statement, (assuming your task will still be to get a single maximum timestamp per user) it's that it will look something like this:
select u.firstname, u.lastname, user_max_time.userName, user_max_time.max_time
from users u,
( select (sometable).userName, max((sometable).(timestamp column))
from (data of interest)
group by (sometable).userName) user_max_time
where u.userName = user_max_time.userName
order by max_time desc;
Your task here would then be to replace the ()s inside the the user_max_time subselect with something that makes sense and maps to your requirements. In terms of a general approach to complex sql, the major suggestion is to build the query from the innermost subselects back out (testing along the way to make sure performance is ok and you don't need intermediate tables).
Anyways, if you're having trouble, and can come back with examples, would be happy to help.
Cheers,
Ben
I guess this query is a little basic and I should know more about SQL but haven't done much with joins yet which I guess is the solution here.
What I have is a table of people and a table of job roles they hold. A person can have multiple jobs and I wish to have one set of results with a row per person containing their details and their job roles.
Two example tables (people and job_roles) are below so you can understand the question easier.
People
id | name | email_address | phone_number
1 | paul | paul#example.com | 123456
2 | bob | bob#example.com | 567891
3 | bart | bart#example.com | 987561
job_roles
id | person_id | job_title | department
1 | 1 | secretary | hr
2 | 1 | assistant | media
3 | 2 | manager | IT
4 | 3 | finance clerk | finance
4 | 3 | manager | IT
so that I can output each person and their roles like such
Name: paul
Email Address: paul#example.com
Phone: 123456
Job Roles:
Secretary for HR department
Assistant for media department
_______
Name: bob
Email address: bob#example.com
Phone: 567891
Job roles:
Manager for IT department
So how would I get each persons information (from the people table) along with their job details (from the job_roles table) to output like the example above. I guess it would be some kind of way of merging their jobs and their relevant departments into a jobs column that can be split up for output, but maybe there is a better way and what would the sql look like?
Thanks
Paul
PS it would be a mySQL database if that makes any difference
It looks like a straight-forward join:
SELECT p.*, j.*
FROM People AS p INNER JOIN Roles AS r ON p.id = r.person_id
ORDER BY p.name;
The remainder of the work is formatting; that's best done by a report package.
Thanks for the quick response, that seems a good start but you get multiple rows per person like (you have to imagine this is a table as you don't seem to be able to format in comments):
id | Name | email_address | phone_number | job_role | department
1 | paul | paul#example.com | 123456 | secretary | HR
1 | paul | paul#example.com | 123456 | assistant | media
2 | bob | bob#example.com | 567891 | manager | IT
I would like one row per person ideally with all their job roles in it if that's possible?
It depends on your DBMS, but most available ones do not support RVAs - relation-valued attributes. What you'd like is to have the job role and department part of the result like a table associated with the user:
+----+------+------------------+--------------+------------------------+
| id | Name | email_address | phone_number | dept_role |
+----+------+------------------+--------------+------------------------+
| | | | | +--------------------+ |
| | | | | | job_role | dept | |
| 1 | paul | paul#example.com | 123456 | | secretary | HR | |
| | | | | | assistant | media | |
| | | | | +--------------------+ |
+----+------+------------------+--------------+------------------------+
| | | | | +--------------------+ |
| | | | | | job_role | dept | |
| 2 | bob | bob#example.com | 567891 | | manager | IT | |
| | | | | +--------------------+ |
+----+------+------------------+--------------+------------------------+
This accurately represents the information you want, but is not usually an option.
So, what happens next depends on your report generation tool. Using the one I'm most familiar with, (Informix ACE, part of Informix SQL, available from IBM for use with the Informix DBMSs), you would simply ensure that the data is sorted and then print the name, email address and phone number in the 'BEFORE GROUP OF id' section of the report, and in the 'ON EVERY ROW' section you would process (print) just the role and department information.
It is often a good idea to separate the report formatting from the data retrieval operations; this is an example of where it is necessary unless your DBMS has unusual features to help with the formatting of selected data.
Oh dear that sounds very complicated and not something I could run easily on a mySQL database in a PHP page?
The RVA stuff - you're right, that is not for MySQL and PHP.
On the other hand, there are millions of reports (meaning results from queries that are formatted for presentation to a user) that do roughly this. The technical term for them is 'Control-Break Report', but the basic idea is not hard.
You keep a record of the 'id' number you last processed - you can initialize that to -1 or 0.
When the current record has a different id number from the previous number, then you have a new user and you need to start a new set of output lines for the new user and print the name, email address and phone number (and change the last processed id number). When the current record has the same id number, then all you do is process the job role and department information (not the name, email address and phone number). The 'break' occurs when the id number changes. With a single level of control-break, it is not hard; if you have 4 or 5 levels, you have to do more work, and that's why there are reporting packages to handle it.
So, it is not hard - it just requires a little care.
RE:
I was hoping SQL could do something
clever and join the rows together
nicely so I had essentially a jobs
column with that persons jobs in it.
You can get fairly close with
SELECT p.id, p.name, p.email_address, p.phone_number,
group_concat(concat(job_title, ' for ', department, ' department') SEPARATOR '\n') AS JobRoles
FROM People AS p
INNER JOIN job_roles AS r ON p.id = r.person_id
GROUP BY p.id, p.name, p.email_address, p.phone_number
ORDER BY p.name;
Doing it the way you're wanting would mean the result set arrays could have infinite columns, which would be very messy. for example, you could left join the jobs table 10 times and get job1, job2, .. job10.
I would do a single join, then use PHP to check if the name ID is the same from 1 row to the next.
One way might be to left outer join the tables and then load them up into an array using
$people_array =array();
while($row1=mysql_fetch_assoc($extract1)){
$people_array[] = $row1;
}
and then loop through using
for ($x=0;$x<=sizeof($people_array;)
{
echo $people_array[$x][id];
echo $people_array[$x][name];
for($y=0;$y<=$number_of_roles;$y++)
{
echo $people_array[$x][email_address];
echo $people_array[$x][phone_number];
$x++;
}
}
You might have to play with the query a bit and the loops but it should do generally what you want.For it to work as above every person would have to have the same number of roles, but you may be able to fill in the blanks in your table