I have data that has a nested mapping in one of the fields. For example, the data is in a table called 'customers' and looks like this:
Name: Bill Jones
Address: {"billing":{"street":"123 Main", "city":"Chicago", "state":"IL"},"shipping":{"street":"432 Copper", "city":"New York", "state":"NY"}}
What sort of query can I write to explode the entire dataset out?
I started with the following:
select name, key1, value1
from customers
lateral view explode(address) table1 as key1, value1
That got me part of the way there. It splits up the "billing" and "shipping" fields. However, I can't explode out the remaining. I tried this, but got an error message:
select name, key1, key2, value2
from customers
lateral view explode(address) table1 as key1, value1
lateral view explode(value1) table2 as key2, value2
I know I'm not doing it right, but not sure what the fix is?
Thanks,
Chetan
Demo
create table customers (Name string, addresses map<string,struct<street1:string,street2:string,city:string,state:string>>);
insert into customers
select 'Bill Jones'
,map
(
'billing' ,named_struct('street1','123 Main' ,'street2','' ,'city','Chicago' ,'state','IL')
,'shipping' ,named_struct('street1','432 Copper' ,'street2','' ,'city','New York' ,'state','NY')
)
;
Option 1
select name
,addresses['billing'].street1 as billing_street1
,addresses['billing'].street2 as billing_street2
,addresses['billing'].city as billing_city
,addresses['billing'].state as billing_state
,addresses['shipping'].street1 as shipping_street1
,addresses['shipping'].street2 as shipping_street2
,addresses['shipping'].city as shipping_city
,addresses['shipping'].state as shipping_state
from customers
;
+------------+-----------------+-----------------+--------------+---------------+------------------+------------------+---------------+----------------+
| name | billing_street1 | billing_street2 | billing_city | billing_state | shipping_street1 | shipping_street2 | shipping_city | shipping_state |
+------------+-----------------+-----------------+--------------+---------------+------------------+------------------+---------------+----------------+
| Bill Jones | 123 Main | | Chicago | IL | 432 Copper | | New York | NY |
+------------+-----------------+-----------------+--------------+---------------+------------------+------------------+---------------+----------------+
Option 2
select name
,key as address_type
,value.street1
,value.street2
,value.city
,value.state
from customers
lateral view explode(addresses) a
;
+------------+--------------+------------+---------+----------+-------+
| name | address_type | street1 | street2 | city | state |
+------------+--------------+------------+---------+----------+-------+
| Bill Jones | billing | 123 Main | | Chicago | IL |
| Bill Jones | shipping | 432 Copper | | New York | NY |
+------------+--------------+------------+---------+----------+-------+
Based on your comments, you have the following structure
name (string), addresses (map>), email (string), phone (string), spend (int)
What you have here, is a map of string, structure not a map of maps, so you can query the data in the folllowing way
select name, key1,
value1.street1,
value1.street2,
value1.city,
value1.state
from customers
lateral view explode(address) table1 as key1, value1
Related
I have three tables: ACCT, PERS, ORG. Each ACCT is owned by either a PERS or ORG. The PERS and ORG tables are very similar and so are all of their child tables, but all PERS and ORG data is separate.
I'm writing a query to get PERS and ORG information for each account in ACCT and I'm curious what the best method of combining the information is. Should I use a series of left joins and NULL functions to fill in the blanks, or should I write the queries separately and use UNION to combine?
I've already written separate queries for PERS ACCT's and another for ORG ACCT's and plan on using UNION. My question more pertains to best practice in the future.
I'm expecting both to give me my desired my results, but I want to find the most efficient method both in development time and run time.
EDIT: Sample Table Data
ACCT Table:
+---------+---------+--------------+-------------+
| ACCTNBR | ACCTTYP | OWNERPERSNBR | OWNERORGNBR |
+---------+---------+--------------+-------------+
| 555001 | abc | 3010 | |
| 555002 | abc | | 2255 |
| 555003 | tre | 5125 | |
| 555004 | tre | 4485 | |
| 555005 | dsa | | 6785 |
+---------+---------+--------------+-------------+
PERS Table:
+---------+--------------+---------------+----------+-------+
| PERSNBR | PHONE | STREET | CITY | STATE |
+---------+--------------+---------------+----------+-------+
| 3010 | 555-555-5555 | 1234 Main St | New York | NY |
| 5125 | 555-555-5555 | 1234 State St | New York | NY |
| 4485 | 555-555-5555 | 6542 Vine St | New York | NY |
+---------+--------------+---------------+----------+-------+
ORG Table:
+--------+--------------+--------------+----------+-------+
| ORGNBR | PHONE | STREET | CITY | STATE |
+--------+--------------+--------------+----------+-------+
| 2255 | 222-222-2222 | 1000 Main St | New York | NY |
| 6785 | 333-333-3333 | 400 4th St | New York | NY |
+--------+--------------+--------------+----------+-------+
Desired Output:
+---------+---------+--------------+-------------+--------------+---------------+----------+-------+
| ACCTNBR | ACCTTYP | OWNERPERSNBR | OWNERORGNBR | PHONE | STREET | CITY | STATE |
+---------+---------+--------------+-------------+--------------+---------------+----------+-------+
| 555001 | abc | 3010 | | 555-555-5555 | 1234 Main St | New York | NY |
| 555002 | abc | | 2255 | 222-222-2222 | 1000 Main St | New York | NY |
| 555003 | tre | 5125 | | 555-555-5555 | 1234 State St | New York | NY |
| 555004 | tre | 4485 | | 555-555-5555 | 6542 Vine St | New York | NY |
| 555005 | dsa | | 6785 | 333-333-3333 | 400 4th St | New York | NY |
+---------+---------+--------------+-------------+--------------+---------------+----------+-------+
Query Option 1: Write 2 queries and use UNION to combine them:
select a.acctnbr, a.accttyp, a.ownerpersnbr, a.ownerorgnbr, p.phone, p.street, p.city, p.state
from acct a
inner join pers p on p.persnbr = a.ownerpersnbr
UNION
select a.acctnbr, a.accttyp, a.ownerpersnbr, a.ownerorgnbr, o.phone, o.street, o.city, o.state
from acct a
inner join org o on o.orgnbr = a.ownerorgnbr
Option 2: Use NVL() or Coalesce to return a single data set:
SELECT a.acctnbr,
a.accttyp,
NVL(a.ownerpersnbr, a.ownerorgnbr) Owner,
NVL(p.phone, o.phone) Phone,
NVL(p.street, o.street) Street,
NVL(p.city, o.city) City,
NVL(p.state, o.state) State
FROM
acct a
LEFT JOIN pers p on p.persnbr = a.ownerpersnbr
LEFT JOIN org o on o.orgnbr = a.ownerorgnbr
There are way more fields in each of the 3 tables as well as many more PERS and ORG tables in my actual query. Is one way better (faster, more efficient) than another?
That depends, on what you consider "better".
Assuming, that you will always want to pull all rows from ACCT table, I'd say to go for the LEFT OUTER JOIN and no UNION. (If using UNION, then rather go for UNION ALL variant.)
EDIT: As you've already shown your queries, mine is no longer required, and did not match your structures. Removing this part.
Why LEFT JOIN? Because with UNION you'd have to go through ACCT twice, based on "parent" criteria (whether separate or done INNER JOIN criteria), while with plain LEFT OUTER JOIN you'll probably get just one pass through ACCT. In both cases, rows from "parents" will most probably be accessed based on primary keys.
As you are probably considering performance, when looking for "better", as always: Test your queries and look at the execution plans with adequate and fresh database statistics in place, as depending on the data "layout" (histograms, etc.) the "better" may be something completely different.
I think you misunderstand what a Union does versus a join statement. A union takes the records from multiple tables, generally similar or the same structure and combines them into a single resultset. It is not meant to combine multiple dissimilar tables.
What I am seeing is that you have two tables PERS and ORG with some of the same data in it. In this case I suggest you union those two tables and then join to ACCT to get the sample output.
In this case to get the output as you have shown you would want to use Outer joins so that you don't drop any records without a match. That will give you nulls in some places but most of the time that is what you want. It is much easier to filter those out later.
Very rough sample code.
SELECT a.*, b.*
from Acct as a
FULL OUTER JOIN (
Select * from PERS UNION Select * from ORG
) as b
ON a.ID = b.ID
I have a table with data stored vertically, I have shown a simplified example below which has a record for each city a customer has lived in:
| CUSTOMER | CITY |
------------------------------
| John | London |
| John | Manchester |
| Sarah | Cardiff |
| Sarah | Edinburgh |
| Sarah | Liverpool |
| Craig | Manchester |
| Craig | London |
I am trying to come up with an SQL query that will return all unique combinations of cities so in the example above, John and Craig have both lived in London and Manchester but Sarah has lived in different cities (Cardiff, Edinburgh and Liverpool) so I would like an output as below (which can handle any amount of cities)
| CITY1 | CITY2 | CITY3 |
--------------------------------------------
| London | Manchester | |
| Cardiff | Edinburgh | Liverpool |
I have tried using a crosstab query to view the data horizontally like this:
TRANSFORM Max(City)
SELECT Customer
FROM tblCities
GROUP BY Customer
PIVOT City
but it is just returning a field for all cities for every customer. Does anyone know if this is possible using SQL?
p.s Ideally it will ignore the order of cities
This was a nice challenge! The query below gets the groupings per customer. It doesn't discard the duplicates where multiple customers have lived in the same combination of cities ... I'll let you or others find a way to handle that.
TRANSFORM Min(OrderedList.City) AS MinOfCity
SELECT OrderedList.Customer
FROM (SELECT CustomerCities.Customer, CustomerCities.City, Count(1) AS CityNo
FROM CustomerCities INNER JOIN CustomerCities AS CustomerCities_1 ON CustomerCities.Customer = CustomerCities_1.Customer
WHERE (((CustomerCities.City)>=[CustomerCities_1].[City]))
GROUP BY CustomerCities.Customer, CustomerCities.City) OrderedList
GROUP BY OrderedList.Customer
PIVOT "CITY" & [CityNo];
Is this what you want?
select distinct c1.city, c2.city
from tblCities as c1 inner join
tblCities as c2
on c1.customer = c2.customer and c1.city < c2.city;
This returns all pairs of cities that appear for any single customer.
Here is a query which might work assuming each customer is only associated with two cities:
SELECT DISTINCT t.city_1, t.city_2
FROM
(
SELECT MIN(CITY) AS city_1, MAX(CITY) AS city_2
FROM tblCities
GROUP BY CUSTOMER
) t
Here are the two tables i have: [all columns in both tables are of type "text"], Table name and the column names are in bold fonts.
Names
--------------------------------
Name | DoB | Team |
--------------------------------
Harry | 3/12/85 | England
Kevin | 8/07/86 | England
James | 5/05/89 | England
Scores
------------------------
ScoreName | Score
------------------------
James-1 | 120
Harry-1 | 30
Harry-2 | 40
James-2 | 56
End result i need is a table that has the following
NameScores
---------------------------------------------
Name | DoB | Team | ScoreData
---------------------------------------------
Harry | 3/12/85 | England | "{"ScoreName":"Harry-1", "Score":"30"}, {"ScoreName":"Harry-2", "Score":"40"}"
Kevin | 8/07/86 | England | null
James | 5/05/89 | England | "{"ScoreName":"James-1", "Score":"120"}, {"ScoreName":"James-2", "Score":"56"}"
I need to do this using a single SQL command which i will use to create a materialized view.
I have gotten as far as realising that it will involve a combination of string_agg, JOIN and JSON, but haven't been able to crack it fully. Please help :)
I don't think the join is tricky. The complication is building the JSON object:
select n.name, n.dob, n.team,
json_agg(json_build_object('ScoreName', s.name,
'Score', s.score)) as ScoreData
from names n left join
scores s
ons.name like concat(s.name, '-', '%')
group by n.name, n.dob, n.team;
Note: json_build_object() was introduced in Postgres 9.4.
EDIT:
I think you can add a case statement to get the simple NULL:
(case when s.name is null then NULL
else json_agg(json_build_object('ScoreName', s.name,
'Score', s.score))
end) as ScoreData
Use json_agg() with row_to_json() to aggregate scores data into a json value:
select n.*, json_agg(row_to_json(s)) "ScoreData"
from "Names" n
left join "Scores" s
on n."Name" = regexp_replace(s."ScoreName", '(.*)-.*', '\1')
group by 1, 2, 3;
Name | DoB | Team | ScoreData
-------+---------+---------+---------------------------------------------------------------------------
Harry | 3/12/85 | England | [{"ScoreName":"Harry-1","Score":30}, {"ScoreName":"Harry-2","Score":40}]
James | 5/05/89 | England | [{"ScoreName":"James-1","Score":120}, {"ScoreName":"James-2","Score":56}]
Kevin | 8/07/86 | England | [null]
(3 rows)
Here is the result I need, simplified:
select name, phonenumber
from contacttmp
left outer join phonetmp on (contacttmp.id = phonetmp.contact_id);
name | phonenumber
-------+--------------
bob | 111-222-3333
bob | 111-222-4444
bob | 111-222-5555
frank | 111-222-6666
joe | 111-222-7777
The query, however displays the name, I'm trying to omit the name after the first result:
name | phonenumber
-------+--------------
bob | 111-222-3333
| 111-222-4444
| 111-222-5555
frank | 111-222-6666
joe | 111-222-7777
Here's how I made the example tables and the data:
create table contacttmp (id serial, name text);
create table phonetmp (phoneNumber text, contact_id integer);
select * from contacttmp;
id | name
----+-------
1 | bob
2 | frank
3 | joe
select * from phonetmp ;
phonenumber | contact_id
--------------+------------
111-222-3333 | 1
111-222-4444 | 1
111-222-5555 | 1
111-222-6666 | 2
111-222-7777 | 3
Old part of question
I'm working on a contacts program in PHP and a requirement is to display the results but omit the other fields after the first record is displayed if there are multiple results of that same record.
From the postgres tutorial join examples I'm doing something like this with a left outer join:
SELECT *
FROM weather LEFT OUTER JOIN cities ON (weather.city = cities.name);
city | temp_lo | temp_hi | prcp | date | name | location
--------------+---------+---------+------+------------+---------------+-----------
Hayward | 37 | 54 | | 1994-11-29 | |
San Francisco | 46 | 50 | 0.25 | 1994-11-27 | San Francisco | (-194,53)
San Francisco | 43 | 57 | 0 | 1994-11-29 | San Francisco | (-194,53)
I can't figure out how to, or if it is possible to, alter the above query to not display the other fields after the first result.
For example, if we add the clause "WHERE location = '(-194,53)'" we don't want the second (and third if there is one) results to display the columns other than location, so the query (plus something extra) and the result would look like this:
SELECT *
FROM weather LEFT OUTER JOIN cities ON (weather.city = cities.name)
WHERE location = '(-194,53)';
city | temp_lo | temp_hi | prcp | date | name | location
--------------+---------+---------+------+------------+---------------+-----------
San Francisco | 46 | 50 | 0.25 | 1994-11-27 | San Francisco | (-194,53)
| | | | | | (-194,53)
Is this possible with some kind of JOIN or exclusion or other query? Or do I have to remove these fields in PHP after getting all the results (would rather not do).
To avoid confusion, I'm required to achieve a result set like:
city | temp_lo | temp_hi | prcp | date | name | location
--------------+---------+---------+------+------------+---------------+-----------
San Francisco | 46 | 50 | 0.25 | 1994-11-27 | San Francisco | (-194,53)
| | | | | | (-19,5)
| | | | | | (-94,3)
Philadelphia | 55 | 60 | 0.1 | 1995-12-12 | Philadelphia | (-1,1)
| | | | | | (-77,55)
| | | | | | (-3,33)
Where any additional results for the same record (city) with different locations would only display the different location.
You can do this type of logic in SQL, but it is not recommended. The result set from SQL queries is in a table format. Tables represented unordered sets and generally have all columns meaning the same thing.
So, having a result set that depends on the values from the "preceding" row is not a proper way to use SQL. Although you can get this result in Postgres, I do not recommend it. Usually, this type of formatting is done on the application side.
If you want to avoid repeating the same information, you can use a window function that tells you the position of that row in the group (a PARTITION for this purpose, not a group in the GROUP BY sense), then hide the text for the columns you don't want to repeat if that position in the group is greater than 1.
WITH joined_results AS (
SELECT
w.city, c.location, w.temp_lo, w.temp_hi, w.prcp, w.date,
ROW_NUMBER() OVER (PARTITION BY w.city, c.location ORDER BY date) AS pos
FROM weather w
LEFT OUTER JOIN cities c ON (w.city = c.name)
ORDER BY w.city, c.location
)
SELECT
CASE WHEN pos > 1 THEN '' ELSE city END,
CASE WHEN pos > 1 THEN '' ELSE location END,
temp_lo, temp_hi, prcp, date
FROM joined_results;
This should give you this:
city | location | temp_lo | temp_hi | prcp | date
---------------+-----------+---------+---------+------+------------
Hayward | | 37 | 54 | | 1994-11-29
San Francisco | (-194,53) | 46 | 50 | 0.25 | 1994-11-27
| | 43 | 57 | 0 | 1994-11-29
To understand what ROW_NUMBER() OVER (PARTITION BY w.city, c.location ORDER BY date) AS pos does, it probably worth looking at what you get with SELECT * FROM joined_results:
city | location | temp_lo | temp_hi | prcp | date | pos
---------------+-----------+---------+---------+------+------------+-----
Hayward | | 37 | 54 | | 1994-11-29 | 1
San Francisco | (-194,53) | 46 | 50 | 0.25 | 1994-11-27 | 1
San Francisco | (-194,53) | 43 | 57 | 0 | 1994-11-29 | 2
After that, just replace what you don't want with white space using CASE WHEN pos > 1 THEN '' ELSE ... END.
(This being said, it's something I'd generally prefer to do in the presentation layer rather than in the query.)
Consider the slightly modified test case in the fiddle below.
Simple case
For the simple case dealing with a single column from each column, comparing to the previous row with the window function lag() does the job:
SELECT CASE WHEN lag(c.contact) OVER (ORDER BY c.contact, p.phone_nr)
= c.contact THEN NULL ELSE c.contact END
, p.phone_nr
FROM contact c
LEFT JOIN phone p USING (contact_id);
You could repeat that for n columns, but that's tedious
For many columns
SELECT c.*, p.phone_nr
FROM (
SELECT *
, row_number() OVER (PARTITION BY contact_id ORDER BY phone_nr) AS rn
FROM phone
) p
LEFT JOIN contact c ON c.contact_id = p.contact_id AND p.rn = 1;
Something like a "reverse LEFT JOIN". This is assuming referential integrity (no missing rows in contact. Also, contacts without any entries in phone are not in the result. Easy to add if need should be.
SQL Fiddle.
Aside, your query in the first example exhibits a rookie mistake.
SELECT * FROM weather LEFT OUTER JOIN cities ON (weather.city = cities.name)
WHERE location = '(-194,53)';
One does not combine a LEFT JOIN with a WHERE clause on the right table. Doesn't makes sense. Details:
Explain JOIN vs. LEFT JOIN and WHERE condition performance suggestion in more detail
Except to test for existence ...
Select rows which are not present in other table
I have three tables like this:
Person table:
person_id | name | dob
--------------------------------
1 | Naveed | 1988
2 | Ali | 1985
3 | Khan | 1987
4 | Rizwan | 1984
Address table:
address_id | street | city | state | country
----------------------------------------------------
1 | MAJ Road | Karachi | Sindh | Pakistan
2 | ABC Road | Multan | Punjab | Pakistan
3 | XYZ Road | Riyadh | SA | SA
Person_Address table:
person_id | address_id
----------------------
1 | 1
2 | 2
3 | 3
Now I want to get all records of Person_Address table but also with their person and address records like this by one query:
person_id| name | dob | address_id | street | city | state | country
----------------------------------------------------------------------------------
1 | Naveed | 1988 | 1 | MAJ Road | Karachi | Sindh | Pakistan
2 | Ali | 1985 | 2 | ABC Road | Multan | Punjab | Pakistan
3 | Khan | 1987 | 3 | XYZ Road | Riyadh | SA | SA
How it is possible using zend? Thanks
The reference guide is the best starting point to learn about Zend_Db_Select. Along with my example below, of course:
//$db is an instance of Zend_Db_Adapter_Abstract
$select = $db->select();
$select->from(array('p' => 'person'), array('person_id', 'name', 'dob'))
->join(array('pa' => 'Person_Address'), 'pa.person_id = p.person_id', array())
->join(array('a' => 'Address'), 'a.address_id = pa.address_id', array('address_id', 'street', 'city', 'state', 'country'));
It's then as simple as this to fetch a row:
$db->fetchRow($select);
In debugging Zend_Db_Select there's a clever trick you can use - simply print the select object, which in turn invokes the toString method to produce SQl:
echo $select; //prints SQL
I'm not sure if you're looking for SQL to do the above, or code using Zend's facilities. Given the presence of "sql" and "joins" in the tags, here's the SQL you'd need:
SELECT p.person_id, p.name, p.dob, a.address_id, street, city, state, country
FROM person p
INNER JOIN Person_Address pa ON pa.person_id = p.person_id
INNER JOIN Address a ON a.address_id = pa.address_id
Bear in mind that the Person_Address tells us that there's a many-to-many relationship between a Person and an Address. Many Persons may share an Address, and a Person may have more than one address.
The SQL above will show ALL such relationships. So if Naveed has two Address records, you will have two rows in the result set with person_id = 1.