From this Question we learned to use a subquery to find information once-removed.
Subquery we learned :
SELECT * FROM papers WHERE writer_id IN ( SELECT id FROM writers WHERE boss_id = 4 );
Now, I need to search a table, both in column values that table, and in column values related by id on another table.
Here are the same tables, but col values contain more text for our "searching" reference...
writers :
id
name
boss_id
1
John Jonno
2
2
Bill Bosworth
2
3
Andy Seaside
4
4
Hank Little
4
5
Alex Crisp
4
The writers have papers they write...
papers :
id
title
writer_id
1
Boston
1
2
Chicago
4
3
Cisco
3
4
Seattle
2
5
North
5
I can use this to search only the names on writers...
Search only writers.name : (Not what I want to do)
SELECT * FROM writers WHERE LOWER(name) LIKE LOWER('%is%');
Output for above search : (Not what I want to do)
id
name
boss_id
5
Alex Crisp
4
I want to return cols from writers (not papers), but searching text both in writers.name and the writers.id-associated papers.title.
For example, if I searched "is", I would get both:
Alex Crisp (for 'is' in the name 'Crisp')
Andy Seaside (because Andy wrote a paper with 'is' in the title 'Cisco')
Output for "is" search :
id
title
writer_id
2
Chicago
4
4
Seattle
2
Here's what I have that doesn't work:
SELECT * FROM papers WHERE LOWER(title) LIKE LOWER('%is%') OR writer_id ( writers=writer_id WHERE LOWER(name) LIKE LOWER('%$is%') );
The best way to express this criteria is by using a correlated query with exists:
select *
from writers w
where Lower(w.name) like '%is%'
or exists (
select * from papers p
where p.writer_id = w.id and Lower(p.title) like '%is%'
);
Note you don't need to use lower on the string you are providing, and you should only use lower if your collation truly is case-sensitive as using the function makes the search predicate unsargable.
Since you want to return cols from writers (not papers) you should select them first, and use stuff from papers in the criteria
select *
from writers w
where
w.name like '%is%'
or
w.id in (select p.writer_id
paper p
where p.title like '%is%'
)
You can add your LOWER functions (my sql environment is not case-sensitive, so I didn't need them)
Related
I need to create automatic weekly assignments of items to sites for my employees.
The items table items_bank looks like that(of course there will be a lot of items with few more languages) :
**item_id** **item_name** **language**
1 Jorge Garcia English
2 Chrissy Metz English
3 Nina Hagen German
4 Harald Glööckle German
5 Melissa Anderson French
6 Pauley Perrette French
My second table is the sites table :
**site_id** **site_name**
1 DR
2 LI
3 IG
I need to assign every week items to the sites with the following constraints :
For each site assign at least X items of English, Y items of German, and so on...
we want to create diversity - so we would like to avoid repeating the assignments of the 2 weeks before
I think we need to create another table in which we can save there the history of the last 2 weeks' assignments.
right now I managed to create an SQL query that assigns items but I don't know how to take the constraints under consideration this is what I create so far :
WITH numbered_tasks AS (
SELECT t.*, row_number() OVER (ORDER BY rand()) item_number, count(*) OVER () total_items
FROM item_bank t
),
numbered_employees AS (
SELECT e.*,row_number() OVER (ORDER BY rand()) site_number,
count(*) OVER () total_sites
FROM sites_bank e
)
SELECT nt.item_name,
ne.acronym
FROM numbered_tasks nt
INNER JOIN numbered_employees ne
ON ne.site_number-1 = mod(nt.item_number-1, ne.total_sites)
Expected results are for the example which says :
site_id=1 have to get 1 item with the English language
site_id=2 have to get 1 item with the German language
site_id=1 have to get 1 item with the French language
**item_id** **language** **Week_number** **site**
1 English 1 1
4 German 1 2
5 French 1 3
Any help will be appreciated!
I have the following table
Id Author
1 Alexander Mccall Smith
2 Ernest Hemingway
3 Giacomo Leopardi
4 Henry David Thoreau
5 Mary Higgins Clark
6 Rabindranath Tagore
7 Thomas Pynchon
8 Zora Neale Hurston
9 William S. Burroughs
10 Virginia Woolf
11 William tell
I want to search the Author by putting first few characters of the first and last name.
eg: Search Text: Will tel
Then the search result show the following result
William tell
eg: Search Text: will Burrou
Then the search result show the following result
William S. Burroughs
eg: Search Text: Will
Then the search result show the following result
William S. Burroughs
William tell
What is the efficient way to achieve this in sql server ?
As you mentioned this can be achieved using Full Text Search. You have to create the FTS catalog and then index on the table and column(s). You stated in the title 'Columns' but I only see one table column in your example so I will create the queries using that.
-- example 1 searching on Will and Tel
SELECT Id, Author
FROM Authors
WHERE CONTAINS(Author, '"Will*" AND "tel*"')
-- example 2 searching on Will and Burrou
SELECT Id, Author
FROM Authors
WHERE CONTAINS(Author, '"will*" AND "Burrou*"')
-- example 3 searching on Will
SELECT Id, Author
FROM Authors
WHERE CONTAINS(Author, '"will*"')
For further reference see
The Contains clause which searches for precise or fuzzy matches.
Article Query with Full-Text Search.
Less efficient than #Igor's answer as the table size grows, but you can also use the Like statement.
The LIKE operator is used in a WHERE clause to search for a specified pattern in a column.
-- example 1 searching on Will and Tel
SELECT Id, Author
FROM Authors
WHERE Author Like('Will%Tel%')
-- example 2 searching on Will and Burrou
SELECT Id, Author
FROM Authors
WHERE Author Like('Will%Burrou%')
-- example 3 searching on Will
SELECT Id, Author
FROM Authors
WHERE Author Like('Will%')
Cons: It is slower than the contains statement.You need to include the % sign after any other keyword you're looking to search for.
Pros: Can be faster than contains statement in cases of smaller(<1000) row tables.
I am investigating the benefits of Oracle Text search, and currently am looking at collecting search text data from multiple (related) tables and storing the data in the smaller table in a 1-to-many relationship.
Consider these 2 simple tables, house and inhabitants, and there are NEVER any uninhabited houses:
HOUSE
ID Address Search_Text
1 44 Some Road
2 31 Letsby Avenue
3 18 Moon Crescent
INHABITANT
ID House Name Nickname
1 1 Jane Doe Janey
2 1 John Doe JD
3 2 Jo Smythe Smithy
4 2 Percy Plum PC
5 3 Apollo Lander Moony
I want to to write SQL that updates the HOUSE.Search_Text column with text from INHABITANT. Now because this is a 1-to-many, the SQL needs to collate the data in INHABITANT for each matching row in house, and then combine the data (comma separated) and update the Search_Text field.
Once done, the Oracle Text search index on HOUSE.Search_Text will return me HOUSEs that match the search criteria, and I can look up INHABITANTs accordingly.
Of course, this is a very simplified example, I want to pick up data from many columns and Full Text Search across fields in both tables.
With the help of a colleague we've got:
select id, ADDRESS||'; '||Names||'; '||Nicknames as Search_Text
from house left join(
SELECT distinct house_id,
LISTAGG(NAME, ', ') WITHIN GROUP (ORDER BY NAME) OVER (PARTITION BY house_id) as Names,
LISTAGG(NICKNAME, ', ') WITHIN GROUP (ORDER BY NICKNAME) OVER (PARTITION BY house_id) as Nicknames
FROM INHABITANT)
i on house.id = i.house_id;
which returns:
1 44 Some Road; Jane Doe, John Doe; JD, Janey
2 31 Letsby Avenue; Jo Smythe, Percy Plum; PC, Smithy
3 18 Moon Crescent; Apollo Lander; Moony
Some questions:
Is this an efficient query to return this data? I'm slightly
concerned about the distinct.
Is this the right way to use Oracle Text search across multiple text fields?
How to update House.Search_Text with the results above? I think I need a correlated subquery, but can't quite work it out.
Would it be more efficient to create a new table containing House_ID and Search_Text only, rather than update House?
I need to modify a SQL table to group slightly mismatched names, and assign all elements in the group a standardized name.
For instance, if the initial table looks like this:
Name
--------
Jon Q
John Q
Jonn Q
Mary W
Marie W
Matt H
I would like to create a new table or add a field to the existing one like this:
Name | StdName
--------------------
Jon Q | Jon Q
John Q | Jon Q
Jonn Q | Jon Q
Mary W | Mary W
Marie W | Mary W
Matt H | Matt H
In this case, I've chosen the first name to assign as the "standardized name," but I don't actually care which one is chosen -- ultimately the final "standardized name" will be hashed into a unique person ID. (I'm also open to alternative solutions that go directly to a numerical ID.) I will have birthdates to match on as well, so the accuracy of the name matching doesn't actually need to be all that precise in practice. I've looked into this a bit and will probably use the Jaro-Winkler algorithm (see e.g. here).
If I knew that the names were all in pairs, this would be a relatively easy query, but there can be an arbitrary number of the same name.
I can easily conceptualize how to do this query in a procedural language, but I'm not very familiar with SQL. Unfortunately I don't have direct access to the data -- it's sensitive data and so somebody else (a bureaucrat) has to run the actual query for me. The specific implementation will be SQL Server, but I'd prefer an implementation-agnostic solution.
EDIT:
In response to a comment, I had the following procedural approach in mind. It's in Python, and I replaced the Jaro-Winkler with simply matching on the first letter of the name, for the sake of having a working code example.
nameList = ['Jon Q', 'John Q', 'Jonn Q', 'Mary W', 'Marie W', 'Larry H']
stdList = nameList[:]
# loop over all names
for i1, name1 in enumerate(stdList):
# loop over later names in list to find matches
for i2, name2 in enumerate(stdList[i1+1:]):
# If there's a match, replace latter with former.
if (name1[0] == name2[0]):
stdList[i1+1+i2] = name1
print stdList
The result is ['Jon Q', 'Jon Q', 'Jon Q', 'Mary W', 'Mary W', 'Larry H'].
Just a thought, but you might be able to use the SOUNDEX() function. This will create a value for the names that are similar.
If you started with something like this:
select name, soundex(name) snd,
row_number() over(partition by soundex(name)
order by soundex(name)) rn
from yt;
See SQL Fiddle with Demo. Which would give a result for each row that is similar along with a row_number() so you could return only the first value for each group. For example, the above query will return:
| NAME | SND | RN |
-----------------------
| Jon Q | J500 | 1 |
| John Q | J500 | 2 |
| Jonn Q | J500 | 3 |
| Matt H | M300 | 1 |
| Mary W | M600 | 1 |
| Marie W | M600 | 2 |
Then you could select all of the rows from this result where the row_number() is equal to 1 and then join back to your main table on the soundex(name) value:
select t1.name,
t2.Stdname
from yt t1
inner join
(
select name as stdName, snd, rn
from
(
select name, soundex(name) snd,
row_number() over(partition by soundex(name)
order by soundex(name)) rn
from yt
) d
where rn = 1
) t2
on soundex(t1.name) = t2.snd;
See SQL Fiddle with Demo. This gives a result:
| NAME | STDNAME |
---------------------
| Jon Q | Jon Q |
| John Q | Jon Q |
| Jonn Q | Jon Q |
| Mary W | Mary W |
| Marie W | Mary W |
| Matt H | Matt H |
Assuming you copy and paste the jaro-winkler implementation from SSC (registration required), the following code will work. I tried to build a SQLFiddle for it but it kept going belly up when I was building the schema.
This implementation has a cheat---I'm using a cursor. Generally, cursors are not conducive to performance but in this case, you need to be able to compare the set against itself. There's probably a graceful number/tally table approach to eliminate the declared cursor.
DECLARE #SRC TABLE
(
source_string varchar(50) NOT NULL
, ref_id int identity(1,1) NOT NULL
);
-- Identify matches
DECLARE #WORK TABLE
(
source_ref_id int NOT NULL
, match_ref_id int NOT NULL
);
INSERT INTO
#src
SELECT 'Jon Q'
UNION ALL SELECT 'John Q'
UNION ALL SELECT 'JOHN Q'
UNION ALL SELECT 'Jonn Q'
-- Oops on matching joan to jon
UNION ALL SELECT 'Joan Q'
UNION ALL SELECT 'june'
UNION ALL SELECT 'Mary W'
UNION ALL SELECT 'Marie W'
UNION ALL SELECT 'Matt H';
-- 2 problems to address
-- duplicates in our inbound set
-- duplicates against a reference set
--
-- Better matching will occur if names are split into ordinal entities
-- Splitting on whitespace is always questionable
--
-- Mat, Matt, Matthew
DECLARE CSR CURSOR
READ_ONLY
FOR
SELECT DISTINCT
S1.source_string
, S1.ref_id
FROM
#SRC AS S1
ORDER BY
S1.ref_id;
DECLARE #source_string varchar(50), #ref_id int
OPEN CSR
FETCH NEXT FROM CSR INTO #source_string, #ref_id
WHILE (##fetch_status <> -1)
BEGIN
IF (##fetch_status <> -2)
BEGIN
IF NOT EXISTS
(
SELECT * FROM #WORK W WHERE W.match_ref_id = #ref_id
)
BEGIN
INSERT INTO
#WORK
SELECT
#ref_id
, S.ref_id
FROM
#src S
-- If we have already matched the value, skip it
LEFT OUTER JOIN
#WORK W
ON W.match_ref_id = S.ref_id
WHERE
-- Don't match yourself
S.ref_id <> #ref_id
-- arbitrary threshold, will need to examine this for sanity
AND dbo.fn_calculateJaroWinkler(#source_string, S.source_string) > .95
END
END
FETCH NEXT FROM CSR INTO #source_string, #ref_id
END
CLOSE CSR
DEALLOCATE CSR
-- Show me the list of all the unmatched rows
-- plus the retained
;WITH MATCHES AS
(
SELECT
S1.source_string
, S1.ref_id
, S2.source_string AS match_source_string
, S2.ref_id AS match_ref_id
FROM
#SRC S1
INNER JOIN
#WORK W
ON W.source_ref_id = S1.ref_id
INNER JOIN
#SRC S2
ON S2.ref_id = W.match_ref_id
)
, UNMATCHES AS
(
SELECT
S1.source_string
, S1.ref_id
, NULL AS match_source_string
, NULL AS match_ref_id
FROM
#SRC S1
LEFT OUTER JOIN
#WORK W
ON W.source_ref_id = S1.ref_id
LEFT OUTER JOIN
#WORK S2
ON S2.match_ref_id = S1.ref_id
WHERE
W.source_ref_id IS NULL
and s2.match_ref_id IS NULL
)
SELECT
M.source_string
, M.ref_id
, M.match_source_string
, M.match_ref_id
FROM
MATCHES M
UNION ALL
SELECT
M.source_string
, M.ref_id
, M.match_source_string
, M.match_ref_id
FROM
UNMATCHES M;
-- To specifically solve your request
SELECT
S.source_string AS Name
, COALESCE(S2.source_string, S.source_string) As StdName
FROM
#SRC S
LEFT OUTER JOIN
#WORK W
ON W.match_ref_id = S.ref_id
LEFT OUTER JOIN
#SRC S2
ON S2.ref_id = W.source_ref_id
query output 1
source_string ref_id match_source_string match_ref_id
Jon Q 1 John Q 2
Jon Q 1 JOHN Q 3
Jon Q 1 Jonn Q 4
Jon Q 1 Joan Q 5
june 6 NULL NULL
Mary W 7 NULL NULL
Marie W 8 NULL NULL
Matt H 9 NULL NULL
query output 2
Name StdName
Jon Q Jon Q
John Q Jon Q
JOHN Q Jon Q
Jonn Q Jon Q
Joan Q Jon Q
june june
Mary W Mary W
Marie W Marie W
Matt H Matt H
There be dragons
Over on SuperUser, I talked about my experience matching people. In this section, I'll list some things to be aware of.
Speed
As part of your matching, hooray in that you have a birthday to augment the match process. I would actually propose you generate a match based exclusively on birthdate first. That is an exact match and one that, with a proper index, SQL Server will be able to quickly include/exclude rows. Because you're going to need it. The TSQL implementation is dog slow. I've been running the equivalent match against a dataset of 28k names (names that had been listed as conference attendees). There ought to be some good overlap there and while I did fill #src with data, it is a table variable with all that that implies but it's been running now for 15 minutes and still hasn't completed.
It's slow for a number of reasons but things that jumped out at me are all the looping and string manipulation in the functions. That is not where SQL Server shines. If you have a need to do a lot of this, it might be a good idea to convert them into CLR methods so at least you can leverage the strength of the .NET libraries for some of the manipulations.
One of the matches we used to use was the Double Metaphone and it would generate a pair of possible phonetic interpretations of the name. Instead of computing that every time, compute it once and store it alongside the name. That would help speed some of the matching. Unfortunately, it doesn't look like JW lends itself to breaking it down like that.
Look at iterating too. We'd first try the algs that we knew were fast. 'John' = 'John' so there's no need to pull out the big guns so we'd try a first pass of straight name checks. If we didn't find a match, we'd try harder. The hope was that by taking various swipes at matching we'd get the low hanging fruit as fast as possible and worry about the harder matches later.
Names
In my SU answer and in the code comments, I mention nicknames. Bill and Billy are going to match. Billy, Liam and William are definitely not going to match even though they may be the same person. You might want to look at a list like this to provide translation between nickname and full name. After running a set of matches on the supplied name, maybe we'd try looking for a match based on the possible root name.
Obviously, there are draw backs to this approach. For example, my grandfather-in-law is Max. Just Max. Not Maximilian, Maximus or any other things you might thing.
Your supplied names look like it's first and last concatenated together. Future readers, if you ever have the opportunity to capture individual portions of a name, please do so. There are products out there that will split names and try to match them up against directories to try and guess whether something is first/middle name or a surname but then you have people like "Robar Mike". If you saw that name there, you'd think Robar is a last name and you'd also pronounce it like "robber." Instead, Robar (say it with a French accent) is his first name and Mike is his last name. At any rate, I think you'll have a better matching experience if you can split first and last out into separate fields and match the individual pieces together. An exact last name match plus a partial first name match might suffice, especially in cases where legally they are "Franklin Roosevelt" and you have a candidate of "F. Roosevelt" Perhaps you have a rule that an initial letter can match. Or you don't.
Noise - as referenced in the JW post and my answer, strip out crap (punctuation, stop words, etc) for matching purposes. Also watch out for honorific tites (phd, jd, etc) and generationals (II, III, JR, SR). Our rule was a candidate with/without a generational could match one in the opposite state (Bob Jones Jr == Bob Jones) or could exactly match the generation (Bob Jones Sr = Bob Jones Sr) but you'd never want to match if both records supplied them and they were conflicting (Bob Jones Sr != Bob Jones Jr).
Case sensitivity, always check your database and tempdb to make sure you aren't making case sensitive matches. And if you are, convert everything to upper or lower for purposes of matching but don't ever throw the supplied casing away. Good luck trying to determine whether latessa should be Latessa, LaTessa or something else.
My query is coming up on a hour's worth of processing with no rows returned so I'm going to kill it and turn in. Best of luck, happy matching.
My problem is Play! Framework / JPA specific. But I think it's applicable to general SQL syntax.
Here is a sample query with a simple JOIN:
return Post.find(
"select distinct p from Post p join p.tags as t where t.name = ?", tag
).fetch();
It's simple and works well.
My question is: What if I want to JOIN on more values in the same table?
Example (Doesn't work. It's a pseudo-syntax I created):
return Post.find(
"select distinct p from Post p join p.tags1 as t, p.tags2 as u, p.tags3 as v where t.name = ?, u.name = ?, v.name = ?", tag1, tag2, tag3,
).fetch();
Your programming logic seems okay, but the SQL statement needs some work. Seems you're new to SQL, and as you pointed out, you don't seem to understand what a JOIN is.
You're trying to select data from 4 tables named POST, TAG1, TAG2, and TAG3.
I don't know what's in these tables, and it's hard to give sample SQL statements without that information. So, I'm going to make something up, just for the purposes of discussion. Let's say that table POST has 6 columns, and there's 8 rows of data in it.
P Fname Lname Country Color Headgear
- ----- ----- ------- ----- --------
1 Alex Andrews 1 1 0
2 Bob Barker 2 3 0
3 Chuck Conners 1 5 0
4 Don Duck 3 6 1
5 Ed Edwards 2 4 2
6 Frank Farkle 4 2 1
7 Geoff Good 1 1 0
8 Hank Howard 1 3 0
We'll say that TAG1, TAG2, and TAG3 are lookup tables, with only 2 columns each. Table TAG1 has 4 country codes:
C Name
- -------
1 USA
2 France
3 Germany
4 Spain
Table TAG2 has 6 Color codes:
C Name
- ------
1 Red
2 Orange
3 Yellow
4 Green
5 Blue
6 Violet
Table TAG3 has 4 Headgear codes:
C Name
- -------
0 None
1 Glasses
2 Hat
3 Monacle
Now, when you select data from these 4 tables, for P=6, you're trying to get something like this:
Fname Lname Country Color Headgear
----- ------ ------- ------ -------
Frank Farkle Spain Orange None
First thing, let's look at your WHERE clause:
where t.name = ?, u.name = ?, v.name = ?
Sorry, but using commas like this is a syntax error. Normally you only want to find data where all 3 conditions are true; you do this by using AND:
where t.name=? AND u.name=? AND v.name=?
Second, why are you joining tables together? Because you need more information. Table POST says that Frank's COUNTRY value is 4; table TAG1 says that 4 means Spain. So we need to "join" these tables together.
The ancient (before 1980, I think) way to join tables is to list more than one table name in the FROM clause, separated by commas. This gives us:
SELECT P.FNAME, P.LNAME, T.NAME As Country, U.NAME As Color, V.NAME As Headgear
FROM POST P, TAG1 T, TAG2 U, TAG3 V
The trouble with this query is that you're not telling it WHICH rows you want, or how they relate to each other. So the database generates something called a "Cartesian Product". It's extremely rare that you want a Cartesian Product - normally this is a HUGE MISTAKE. Even though your database only has 22 rows in it, this SELECT statement is going to return 768 rows of data:
Alex Andrews USA Red None
Alex Andrews USA Red Glasses
Alex Andrews USA Red Hat
Alex Andrews USA Red Monacle
Alex Andrews USA Orange None
Alex Andrews USA Orange Glasses
...
Hank Howard Spain Violet Monacle
That's right, it returns every possible combination of data from the 4 tables. Imagine for a second that the POST table eventually grows to 20000 rows, and the three TAG tables have 100 rows each. The whole database would be less than a megabyte, but the Cartesian Product would have 20,000,000,000 rows of data -- probably about 120 GB of data. Any database engine would choke on that.
So if you want to use the Ancient way of specifying tables, it is VERY IMPORTANT to make sure that your WHERE clause shows the relationship between every table you're querying. This makes a lot more sense:
SELECT P.FNAME, P.LNAME, T.NAME As Country, U.NAME As Color, V.NAME As Headgear
FROM POST P, TAG1 T, TAG2 U, TAG3 V
WHERE P.Country=T.C AND P.Color=U.C AND P.Headgear=V.C
This only returns 8 rows of data.
Using the Ancient way, it's easy to accidentally create Cartesian Products, which are almost always bad. So they revised SQL to make it harder to do. That's the JOIN keyword. Now, when you specify additional tables you can specify how they relate at the same time. The New Way is:
SELECT P.FNAME, P.LNAME, T.NAME As Country, U.NAME As Color, V.NAME As Headgear
FROM POST P
INNER JOIN TAG1 T ON P.Country=T.C
INNER JOIN TAG2 U ON P.Color=U.C
INNER JOIN TAG3 V ON P.Headgear=V.C
You can still use a WHERE clause, too.
SELECT P.FNAME, P.LNAME, T.NAME As Country, U.NAME As Color, V.NAME As Headgear
FROM POST P
INNER JOIN TAG1 T ON P.Country=T.C
INNER JOIN TAG2 U ON P.Color=U.C
INNER JOIN TAG3 V ON P.Headgear=V.C
WHERE P.P=?
If you call this and pass in the value 6, you get only one row back:
Fname Lname Country Color Headgear
----- ------ ------- ------ --------
Frank Farkle Spain Orange None
As was mentioned in the comments, you are looking for an ON clause.
SELECT * FROM TEST1
INNER JOIN TEST2 ON TEST1.A = TEST2.A AND TEST1.B = TEST2.B ...
See example usage of join here:
http://en.wikibooks.org/wiki/Java_Persistence/Relationships#Join_Fetching