I have combined two different tables together, one side is named DynDom and the other is CATH. I am trying to remove duplicates from that table such as below:
However, if i select distinct Dyndom pdbcode from the table, it returns distinct values of that pdbcode.
and
Based on the pictures above, I commented out the DynDom/CATH columns in the table and ran the query separately for DynDom/CATH and it returned those values accordingly, which is what i need and i was wondering if it's possible for me to use 2 distinct statements to return distinct values of the entire table based on the pdbcode.
Here's my code :
select DISTINCT
cath_dyndom_table_2."DYNDOM_DOMAINID",
cath_dyndom_table_2."DYNDOM_DSTART",
cath_dyndom_table_2."DYNDOM_DEND",
cath_dyndom_table_2."DYNDOM_CONFORMERID",
cath_dyndom_table_2.pdbcode,
cath_dyndom_table_2."DYNDOM_ChainID",
cath_dyndom_table_2.cath_pdbcode,
cath_dyndom_table_2."CATH_BEGIN",
cath_dyndom_table_2."CATH_END"
from
cath_dyndom_table_2
where
pdbcode = '2hun'
order by
cath_dyndom_table_2."DYNDOM_DOMAINID",
cath_dyndom_table_2."DYNDOM_DSTART",
cath_dyndom_table_2."DYNDOM_DEND",
cath_dyndom_table_2.pdbcode,
cath_dyndom_table_2.cath_pdbcode,
cath_dyndom_table_2."CATH_BEGIN",
cath_dyndom_table_2."CATH_END";
In the end, i would like to search domains from DynDom and CATH, based on the pdbcode and return the rows without having duplicate values.
Thank you.
UPDATE :
This is my VIEW table that i have done.
CREATE VIEW cath_dyndom_table AS
SELECT
r.domainid AS "DYNDOM_DOMAINID",
r.DomainStart AS "DYNDOM_DSTART",
r.Domain_End AS "DYNDOM_DEND",
r.ddid AS "DYN_DDID",
r.confid AS "DYNDOM_CONFORMERID",
r.pdbcode,
r.chainid AS "DYNDOM_ChainID",
d.cath_pdbcode,
d.cathbegin AS "CATH_BEGIN",
d.cathend AS "CATH_END"
FROM dyndom_domain_table r
FULL OUTER JOIN cath_domains d ON d.cath_pdbcode::character(4) = r.pdbcode
ORDER BY confid ASC;
What you are getting is the cartesian product of the ´two tables`.
In order to get one line without duplicates you need to have to have a 1-to-1 relation between both tables.
You can see HERE what are cartesian joins and HERE how to avoid them!
It sounds as though you want a UNION of domain name and ranges from each table - this can be achieved like so:
SELECT DYNDOM_DOMAINID, DYNDOM_DSTART, DYNDOM_DEND
FROM DynDom
UNION
SELECT RTRIM(cath_pdbcode), CATH_BEGIN, CATH_END
FROM CATH
This should eliminate exact duplicates (ie. where the domain name, start and end are all identical) but will not eliminate duplicate domain names with different ranges - if these exist you will need to decide how to handle them (retain them as separate entries, combine them with lowest start and highest end, or whatever other option is preferred).
EDIT: Actually, I believe you can get the desired results simply by changing the JOIN ON condition in your view to be:
FULL OUTER JOIN cath_domains d
ON d.cath_pdbcode::character(5) = r.pdbcode || r.chainid AND
r.DomainStart <= d.cathbegin AND
r.Domain_End >= d.cathend
Related
I am trying to compare two lists of unique Household IDs using the Distinct clause. The problem comes when I try to pull in a third column consisting of timestamps into the results.
When I include only the two Household ID columns in the Select statement, the results seem to make sense. I get back two lists of unique IDs.
Here is that query:
select distinct e.household_id, a.hhid
FROM [dbo].[exposure] e
left outer join [dbo].[audience] a
on e.household_id = a.hhid
However, when I just add the "e.imp_ts" column to the Select statement, it looks like SQL completely disregards the Distinct part of the query and pulls in all the duplicate households in the files.
select distinct e.household_id, a.hhid, e.imp_ts
FROM [dbo].[exposure] e
left outer join [dbo].[audience] a
on e.household_id = a.hhid
Can someone please explain why the query doesn't work when I simply add a third column to the Select statement?
Thank you!
It is not that the second query "doesn't work", but rather that it is being asked to provide different results than the first query. As others in the comments have pointed out, because the imp_ts column contains more granular data, the distinct can no longer return a unique list of household IDs. For example, household ID 12345 may contain 5 records, each with unique timestamps on them.
In order to resolve this, you have some choices:
Remove imp_ts from the query.
Return the minimum (most likely first) timestamp
Return the maximum (most likely last) timestamp
For #2 and #3 above, you can use MIN() or MAX() with a GROUP BY to achieve those results. Here is an example of using MIN():
select e.household_id, a.hhid, MIN(e.imp_ts) AS min_imp_ts
FROM [dbo].[exposure] e
left outer join [dbo].[audience] a
on e.household_id = a.hhid
group by e.household_id, a.hhid
I would suggest looking up GROUP BY examples online to get a better idea of what is happening.
I am looking for the best way to combine two tables in a way that will remove duplicate records based on email with a priority of replacing any duplicates with the values in "Table 2", I have considered full outer join and UNION ALL but Union all will be too large as each table has several 1000 columns. I want to create this combination table as my full reference table and save as a view so I can reference it without always adding a union or something to that effect in my already complex statements. From my understanding, a full outer join will not necessarily remove duplicates. I want to:
a. Create table with ALL columns from both tables (fields that don't apply to records in one table will just have null values)
b. Remove duplicate records from this master table based on email field but only remove the table 1 records and keep the table 2 duplicates as they have the information that I want
c. A left-join will not work as both tables have unique records that I want to retain and I would like all 1000+ columns to be retained from each table
I don't know how feasible this even is but thank you so much for any answers!
If I understand your question correctly you want to join two large tables with thousands of columns that (hopefully) are the same between the two tables using the email column as the join condition and replacing duplicate records between the two tables with the records from Table 2.
I had to do something similar a few days ago so maybe you can modify my query for your purposes:
WITH only_in_table_1 AS(
SELECT *
FROM table_1 A
WHERE NOT EXISTS
(SELECT * FROM table_2 B WHERE B.email_field = A.email_field))
SELECT * FROM table_2
UNION ALL
SELECT * FROM only_in_table_1
If the columns/fields aren't the same between tables you can use a full outer join on only_in_table_1 and table_2
try using a FULL OUTER JOIN between the two tables and then a COALESCE function on each resultset column to determine from which table/column the resultset column is populated
I've written a query to filter a table based on criteria found in a master table, and then remove rows that match a third table. I'm executing the query in Access, so I can't use MINUS. It works, but I found that it returns duplicate rows for some, but not all, of the selected records. I fixed it with DISTINCT, but I don't know why it would return duplicates in the first place. It's a pretty simple query:
select distinct sq.*
from
(select List_to_Check.*, Master_List.SELECTION_VAR
from List_to_Check
left join Master_List
on List_to_Check.SUB_ID = Master_List.SUB_ID
where Master_List.SELECTION_VAR = 'criteria'
) as sq
left join List_to_Exclude
on sq.SUB_ID = List_to_Exclude.SUB_ID
where List_to_Exclude.SUB_ID is null
;
Edit: The relationships between all three tables are 1-to-1 on the SUB_ID var. Combined with using a LEFT JOIN, I would expect one line per ID.
I recommend breaking your query apart and checking for duplicates. My guess is that it's your data/ the sub_ID isn't very unique.
Start with you sub query since you're returning all of those columns. If you get duplicates there, your query is going to return duplicates regardless of what is in your exclusion table.
Once you have those duplicates cleared up, check the exclusion table for duplicate sub_Id.
To save time in trouble-shooting, if there are known culprits that are duplicates, you may want to limit the returned values, so you can focus on the peculiarities of those data.
I'm not sure this is a problem, but look into the logic on
on List_to_Check.SUB_ID =
Master_List.SUB_ID
where Master_List.SELECTION_VAR = 'criteria'
Where clauses on data in the right side of a left outer join may not be returning the data you expect. Try this and see what happens:
on List_to_Check.SUB_ID = Master_List.SUB_ID
and Master_List.SELECTION_VAR = 'criteria'
The inner query joins List_to_Check and master but the outer query joins List_to_Exclude with Subscriber(maybe you can change the names i call these 3 tables)
To avoid duplicates you need to use one of the table in both the queries inner and outer. This will avoid duplicates.
I have a table of LoggedDischarges and another table of ActualDischarges.
I am trying to generate a query that will give me all the fields from ActualDischarges excluding those already in LoggedDischarges based on AgencyID, Program and ActivityEndDate
A client can be in multiple programs and be discharged from multiple on the same day. I need to make sure I get LoggedDischarges from each program.
This is what I have but am not sure how to add the other criteria.
select * from ActualDischarges
where (agencychildid ) not in
(select agencyid from LoggedDischarges)
Thank you,
Steve Hathaway
Even if your DBMS supports multiple columns in a subquery like
where (AgencyID, Program, ActivityEndDate) not in
( select AgencyID, Program, ActivityEndDate
from ... )
you better switch to a NOT EXISTS (in case of any NULLs):
select * from ActualDischarges as aD
where NOT EXISTS
(select * from LoggedDischarges as lD
where aD.AgencyID = lD.AgencyID
and aD.Program = lD. Program
and aD.ActivityEndDate= lD.ActivityEndDate)
For this type of match, I would recommend a LEFT JOIN with an IS NULL at the end to determine that the second table does not have the record:
SELECT a.*
FROM ActualDischarges AS a
LEFT JOIN LoggedDischarges AS l
ON agencyid=agencychildid
AND a.program=l.program
AND a.ActivityEndDate=l.ActivityEndDate
WHERE l.agencyid IS NULL
As a side note, definitely avoid using multiple IN statements for situations like this WHERE NOT IN (...) AND NOT IN (...) etc. as you end up excluding records which match different records in LoggedDischarges for different reasons, which is rarely the desired result.
I have a very large view containing 5 million records containing repeated names with each row having unique transaction number. Another view of 9000 records containing unique names is also present. Now I want to retrieve records in first view whose names are present in second view
select * from v1 where name in (select name from v2)
But the query is taking very long to run. Is there any short cut method?
Did you try just using a INNER JOIN. This will return all rows that exist in both tables:
select v1.*
from v1
INNER JOIN v2
on v1.name = v2.name
If you need help learning JOIN syntax, here is a great visual explanation.
You can add the DISTINCT keyword which will remove any duplicate values that the query returns.
use JOIN.
The DISTINCT will allow you to return only unique records from the list since you are joining from the other table and there could be possibilities that a record may have more than one matches on the other table.
SELECT DISTINCT a.*
FROM v1 a
INNER JOIN v2 b
ON a.name = b.name
For faster performance, add an index on column NAME on both tables since you are joining through it.
To further gain more knowledge about joins, kindly visit the link below:
Visual Representation of SQL Joins