I have found a few similar questions to this on SO but nothing which applies to my situation.
I have a large dataset with hundreds of millions of rows in Table 1 and am looking for the most efficient way to run the following query. I am using Google BigQuery but I think this is a general SQL question applicable to any DBMS?
I need to apply an owner to every row in Table 1. I want to join in the following priority:
1: if item_id matches an identifier in Table 2
2: if no item_id matches try match on item_name
3: if no item_id or item_name matches try match on item_division
4: if no item_division matches, return null
Table 1 - Datapoints:
| id | item_id | item_name | item_division | units | revenue
|----|---------|-----------|---------------|-------|---------
| 1 | xyz | pen | UK | 10 | 100
| 2 | pqr | cat | US | 15 | 120
| 3 | asd | dog | US | 12 | 105
| 4 | xcv | hat | UK | 11 | 140
| 5 | bnm | cow | UK | 14 | 150
Table 2 - Identifiers:
| id | type | code | owner |
|----|---------|-----------|-------|
| 1 | id | xyz | bob |
| 2 | name | cat | dave |
| 3 | division| UK | alice |
| 4 | name | pen | erica |
| 5 | id | xcv | fred |
Desired output:
| id | item_id | item_name | item_division | units | revenue | owner |
|----|---------|-----------|---------------|-------|---------|-------|
| 1 | xyz | pen | UK | 10 | 100 | bob | <- id
| 2 | pqr | cat | US | 15 | 120 | dave | <- code
| 3 | asd | dog | US | 12 | 105 | null | <- none
| 4 | xcv | hat | UK | 11 | 140 | fred | <- id
| 5 | bnm | cow | UK | 14 | 150 | alice | <- division
My attempts so far have involved multiple joining the table onto itself and I fear it is becoming hugely inefficient.
Any help much appreciated.
Another option for BigQuery Standard SQL
#standardSQL
SELECT ARRAY_AGG(a)[OFFSET(0)].*,
ARRAY_AGG(owner
ORDER BY CASE
WHEN type = 'id' THEN 1
WHEN type = 'name' THEN 2
WHEN type = 'division' THEN 3
END
LIMIT 1
)[OFFSET(0)] owner
FROM Datapoints a
JOIN Identifiers b
ON (a.item_id = b.code AND b.type = 'id')
OR (a.item_name = b.code AND b.type = 'name')
OR (a.item_division = b.code AND b.type = 'division')
GROUP BY a.id
ORDER BY a.id
It leaves out entries which k=have no owners - like in below result (id=3 is out as it has no owner)
Row id item_id item_name item_division units revenue owner
1 1 xyz pen UK 10 100 bob
2 2 pqr cat US 15 120 dave
3 4 xcv hat UK 11 140 fred
4 5 bnm cow UK 14 150 alice
I am using the following query (thanks #Barmar) but want to know if there is a more efficient way in Google BigQuery:
SELECT a.*, COALESCE(b.owner,c.owner,d.owner) owner FROM datapoints a
LEFT JOIN identifiers b on a.item_id = b.code and b.type = 'id'
LEFT JOIN identifiers c on a.item_name = c.code and c.type = 'name'
LEFT JOIN identifiers d on a.item_division = d.code and d.type = 'division'
I'm not sure if BigQuery optimizes today a query like this - but at least you would be writing a query that gives strong hints to not run the subqueries when not needed:
#standardSQL
SELECT COALESCE(
null
, (SELECT MIN(payload)
FROM `githubarchive.year.2016`
WHERE actor.login=a.user)
, (SELECT MIN(payload)
FROM `githubarchive.year.2016`
WHERE actor.id = SAFE_CAST(user AS INT64))
)
FROM (SELECT '15229281' user) a
4.2s elapsed, 683 GB processed
{"action":"started"}
For example, the following query took a long time to run, but BigQuery could optimize its execution massively in the future (depending on how frequently users needed an operation like this):
#standardSQL
SELECT COALESCE(
"hello"
, (SELECT MIN(payload)
FROM `githubarchive.year.2016`
WHERE actor.login=a.user)
, (SELECT MIN(payload)
FROM `githubarchive.year.2016`
WHERE actor.id = SAFE_CAST(user AS INT64))
)
FROM (SELECT actor.login user FROM `githubarchive.year.2016` LIMIT 10) a
114.7s elapsed, 683 GB processed
hello
hello
hello
hello
hello
hello
hello
hello
hello
hello
Related
I am trying to learn sql.I do some practices.I created a table which called Student.
Id | Name | Amount
1 | Jone | 100
2 | Jack | 200
3 | Emily | 300
4 |Haaland | 500
7 |Ted | 700
I also created Orders table like that:
Id | Name | Amount | Dıscount
1 | Jone | 100 | 10
2 | Jack | 112 | 20
3 | Emily | 300 | 30
4 |Haaland | 500 | 50
5 |Jack | 88 | 12
7 |Ted | 150 | 235
My query is:
select a1.Id Id ,a1.Name Name, a1.Amount Amount , sum(a2.discount)
from student a1
left outer join orders a2
on a1.Id=a2.Id
and a1.Name=a2.Name
and a1.Amount = a2.Amount
group by a1.Id, a1.Name, a1.Amount
Result:
Id | Name | Amount | Dıscount
1 | Jone | 100 | 10
3 | Emily | 300 | 30
4 |Haaland | 500 | 50
2 | Jack | 200 | null
7 | Ted | 700 | null
I get null value for the jack row.I have to use a1.Amount=a2.Amount because I remove amount constraint Ted'discount also appears.
Expected Result :
Id | Name | Amount | Dıscount
1 | Jone | 100 | 10
3 | Emily | 300 | 30
4 |Haaland | 500 | 50
2 | Jack | 200 | 32
7 | Ted |700 | null
I think the logic you want is to pre-aggregate the orders of each name in a subquery, then join by name and amount:
select s.id , s.name, s.amount, o.discount
from student s
left join (
select name, sum(amount) amount, sum(discount) discount
from orders
group by name
) o on o.name = s.name and o.amount = s.amount
What is the confusion? In one row you have:
id name amount
2 Jack 200
And in the other:
id name amount
2 Jack 112
Your join requires equality on all three columns. The amounts don't match, so there is no match for Jack's row and the amount is null.
Your question is not clear on what you actually want to do, so I'll stop here.
The amount for Jack does not match (200 in Student, 88 and 112 in Orders), so nothing can be joined ON a1.Amount = a2.Amount for that record. However, Please be advised that even if one of the values in Amount does match, the GROUP BY function will still not know which Amount you want associated with 'Jack'.
Problem description
Let the tables C and V have those values
>> Table V <<
| UnID | BillID | ProductDesc | Value | ... |
| 1 | 1 | 'Orange Juice' | 3.05 | ... |
| 1 | 1 | 'Apple Juice' | 3.05 | ... |
| 1 | 2 | 'Pizza' | 12.05 | ... |
| 1 | 2 | 'Chocolates' | 9.98 | ... |
| 1 | 2 | 'Honey' | 15.98 | ... |
| 1 | 3 | 'Bread' | 3.98 | ... |
| 2 | 1 | 'Yogurt' | 8.55 | ... |
| 2 | 1 | 'Ice Cream' | 7.05 | ... |
| 2 | 1 | 'Beer' | 9.98 | ... |
| 2 | 2 | 'League of Legends RP' | 40.00 | ... |
>> Table C <<
| UnID | BillID | ClientName | ... |
| 1 | 1 | 'Alexander' | ... |
| 1 | 2 | 'Tom' | ... |
| 1 | 3 | 'Julia' | ... |
| 2 | 1 | 'Tom' | ... |
| 2 | 2 | 'Alexander' | ... |
Table C have the values of each product, which is associated with a bill number. Table V has the relationship between the client name and the bill number. However, the bill number has a counter that is dependent on the UnId, which is the store unity ID. That being said, each store has it`s own Bill number 1, number 2, etc. Also, the number of bills from each store are not equal.
Solution description
I'm trying to make select between the C left join V without sucess. Because each BillID is dependent on the UnID, I have to make the join considering the concatenation between those two columns.
I've used this script, but it gives me an error.
SELECT
SUM(C.Value),
V.ClientName
FROM
C
LEFT JOIN
V
ON
CONCAT(C.UnID, C.BillID) = CONCAT(V.UnID, V.BillID)
GROUP BY
V.ClientName
and SQL server returns me this 'CONCAT' is not a recognized built-in function name.
I'm using Microsoft SQL Server 2008 R2
Is the use of CONCAT wrong? Or is it the way I tried to SELECT? Could you give me a hand?
[OBS: The tables I've present you are just for the purpose of explaining my difficulties. That being said, if you find any errors in the explanation, please let me know to correct them.]
You should be joining on the equality of the UnID and BillID columns in the two tables:
SELECT
c.ClientName,
COALESCE(SUM(v.Value), 0) AS total
FROM C c
LEFT JOIN V v
ON c.UnID = v.UnID AND
c.BillID = v.BillID
GROUP BY
c.ClientName;
In theory you could try joining on CONCAT(UnID, BillID). However, you could run into problems. For example, UnID = 1 with BillID = 23 would, concatenated together, be the same as UnID = 12 and BillID = 3.
Note: We wrap the sum with COALESCE, because should a given client have no entries in the V table, the sum would return NULL, which we then replace with zero.
concat is only available in sql server 2012.
Here's one option.
SELECT
SUM(C.Value),
V.ClientName
FROM
C
LEFT JOIN
V
ON
cast(C.UnID as varchar(100)) + cast(C.BillID as varchar(100)) = cast(V.UnID as varchar(100)) + cast(V.BillID as varchar(100))
GROUP BY
V.ClientName
I have a problem with a query for Oracle with this scenario:
Table People
ID | Name
1 | juan
2 | pedro
3 | luis
Table Properties
ID | nombre_inmueble | FK to Table People
1 | house | 1
2 | garden | 1
3 | terrace | 1
4 | moto | 2
5 | jet | 2
Table Accessories
ID | accessories | FK Table Properties
1 | windows | 1
2 | doors | 1
3 | scale | 2
4 | plants | 3
5 | motor | 4
What I want is only the people who have Properties and that have ALL Accessories, in this case the output would be
1 | juan
What would be the query?
Your query will look like this:
SELECT *
FROM People P
WHERE EXISTS(
SELECT 1
FROM Properties T
WHERE T.PEOPLE= P.ID
)
AND NOT EXISTS(
SELECT 1
FROM Properties T
WHERE T.PEOPLE= P.ID
AND NOT EXISTS(
SELECT 1
FROM Accessories A
WHERE A.Properties = T.ID
)
);
Customer
customer_id | customer_name | customer_city | customer_number
---------------------------------------------------------------
1 | john | sanjose | 978234
2 | chris | newyork | 293
3 | mary | madrid | 342943
4 | tom | bangkok | 8627093
---------------------------------------------------------------
Data
data_id | data_name | data_city | data_number | data_cust_id | customer_id
--------------------------------------------------------------------------------------------
1 | abc | xyz | 990 | 1 | NULL
2 | john | sanjose | 978234 | 1 | NULL
3 | mary | madrid | 8627093 | 3 | NULL
4 | tom | LA | 7729 | 4 | NULL
ActionType
action_id | action_description
-----------------------------------
1 | customer_name
2 | customer_number
3 | customer_city
DataToAction
id | data_id | action_id
--------------------------
1 | 1 | 1
2 | 1 | 2
4 | 2 | 1
5 | 2 | 2
6 | 2 | 3
7 | 3 | 1
8 | 3 | 2
9 | 4 | 1
There are 4 tables -
Customer - Has customer datails
Data - Raw data pulled from an external source (has customer data and others)
ActionType - Has the column names which will be used in a join condition
DataToAction - For each of the raw data row in Data table, the columns to be used in the join is specified here.
Objective - To populate customer_id in 'Data' table.
I need something like this
UPDATE D
SET D.customer_id = C.customer_id
FROM Data D
INNER JOIN Customer C on D.data_cust_id = C.customer_id
WHERE *("GET THE COLUMNS TO BE MATCHED FROM DATATOACTION TABLE AND USE HERE")*
For eg., for Data id 1, i will update customer_id based on customer_name & customer_number, for data id 2 i will udpate customer_id based on customer_name, customer_number & customer_city and so on.
How do I apply the dynamic column conditions in the where clause for each of the row wherein the columns to be matched are specified in a different table.
Well the question is quite unclear. Can u elaborate the final resulset.
Purpose of ActionType table??
UPDATE D
SET D.customer_id = C.customer_id
FROM Data D
INNER JOIN Customer C on D.data_cust_id = C.customer_id
INNER JOIN DataToAction DA ON DA.data_id = D.data_id
I have two tables:
1. Master
| ID | Name | Amount |
|-----|--------|--------|
| 1 | a | 5000 |
| 2 | b | 10000 |
| 3 | c | 5000 |
| 4 | d | 8000 |
2. Detail
| ID |MasterID| PID | Qty |
|-----|--------|-------|------|
| 1 | 1 | 1 | 10 |
| 2 | 1 | 2 | 20 |
| 3 | 2 | 2 | 60 |
| 4 | 2 | 3 | 10 |
| 5 | 3 | 4 | 100 |
| 6 | 4 | 1 | 20 |
| 7 | 4 | 3 | 40 |
I want to select sum(Amount) from Master which joins to Deatil where Detail.PID in (1,2,3)
So I execute the following query:
SELECT SUM(Amount) FROM Master M INNER JOIN Detail D ON M.ID = D.MasterID WHERE D.PID IN (1,2,3)
Result should be 20000. But I am getting 40000
See this fiddle. Any suggestion?
You are getting exactly double the amount because the detail table has two occurences for each of the PIDs in the WHERE clause.
See demo
Use
SELECT SUM(Amount)
FROM Master M
WHERE M.ID IN (
SELECT DISTINCT MasterID
FROM DETAIL
WHERE PID IN (1,2,3) )
What is the requirement of joining the master table with details when you have all your columns are in Master table.
Also, isnt there any FK relationhsip defined on these tables. Looking at your data it seems to me that there should be FK on detail table for MasterId. If that is the case then you do not need join the table at all.
Also, in case you want to make sure that you have records in details table for the records for which you need sum and there is no FK relationship. Then you could give a try for exists instead of join.