How to flatten a json structure in HIVE? - sql

Question

I have a JSON structure that looks like this.
There is a structure nested in a column.
HEADER:
{"user":{"location":"USA","id":1514008171,"name":"Auzzie Jet","screenname":"metalheadgrunge","geoenabled":false},"tweetmessage":"Anthrax - Black - Sunglasses hell","createddate":"2013-06-20T12:08:44","identifier":"1234","geolocation":null}
When I query this, this works:
SELECT *
FROM TBL a
WHERE header.identifier = '1234'
But then, when I want to find the location from the nested json structure. It does not work:
SELECT *
FROM TBL a
WHERE header.identifier = '1234'
and a.header.user.location LIKE '%USA%'
Does anyone know how to query this in HIVE?

Answer 1

For flattening json structure you need to first create lateral view using json_tuple that's how you can be able to achieve what you intend to.
Please find the complete solution how to do it.
Step 1: Create external table tweets with single column tweet with data type of string.
CREATE EXTERNAL table tweets (tweet string);
Now put the json string
{"user":{"location":"USA","id":1514008171,"name":"Auzzie Jet","screenname":"metalheadgrunge","geoenabled":false},"tweetmessage":"Anthrax - Black - Sunglasses hell","createddate":"2013-06-20T12:08:44","identifier":"1234","geolocation":null}
Step 2: In text file named tweets.txt and run the below command to load the data from text file into hive table.
LOAD data local inpath 'tweets.txt' into table tweets;
Once done, now we are ready to play on our Json string.
So basically here what we are trying to achieve is querying on identifier and location fields which are basically at different levels.
user
location : USA
id : 1514008171
name : Auzzie Jet
screenname : metalheadgrunge
geoenabled : false
tweetmessage : Anthrax - Black - Sunglasses hell
createddate : 2013-06-20T12:08:44
identifier : 1234
geolocation : null
Level 1 fields are => user, tweetmessage, createddate,identifier, geolocation
Level 2 Fields are => location, id, name, screenname, geoenabled
So firstly we need to create lateral View on Level 1 so that we can query on Level 1 fields. In our example we need to query on identifier. Also to query on Level 2 fields we need to explode our User view which would be possible by lateral view.
LATERAL VIEW json_tuple(t.tweet, 'user', 'identifier' ) t1 as `user`, `identifier`
and then to query on location, we need to create another lateral view for level 2 fields.
LATERAL VIEW json_tuple(t1.`user`,'name', 'location') t2 as `name`, `location`
and that's it finally we can use select on tweets with lateral views.
Step 3 and Final Query:
SELECT t.* FROM tweets t LATERAL VIEW json_tuple(t.tweet, 'user', 'identifier' ) t1 as `user`, `identifier` LATERAL VIEW json_tuple(t1.`user`,'name', 'location') t2 as `name`, `location` where t1.`identifier`=1234 and `location` ="USA";
For more read on lateral view :LateralView
and on Json_Tuple : JsonTuple