in hive sql I have the following field as date time
date_time
2017-01-01 12:00:00
min_date
2017-02-01 12:00:00
can I compare both fields as date_time > min_date
in my sql query?
how do we compare date time in hive sql?
both timestamp types
You can compare timestamps or strings if they are in sort-able format like this yyyy-MM-dd HH:mm:ss[.f...]
Demo:
hive> select cast('2017-01-01 12:00:00' as timestamp)>cast('2017-02-01 12:00:00' as timestamp);
OK
false
Time taken: 0.13 seconds, Fetched: 1 row(s)
Example with strings:
hive> select '2017-01-01 12:00:00'>'2017-02-01 12:00:00';
OK
false
Time taken: 1.053 seconds, Fetched: 1 row(s)
Related
I'm still learning SQL and I found a couple of solutions on SQL Server or PostgreŅ, but it doesn't seen to work on HUE
DATEDIFF, only allows me to calculate difference between days
seconds, minutes are not available. Help is very welcome.
I was able to split the timestamp with substring_index, but then I can't find the right approach to compare and subtract start_time to end_time in order to obtain the accurate account of seconds. I can't find time functions so I'm assuming I should calculate it based on timestamp. obtained as
from_unixtime(unix_timestamp(start_time, "yyyy-MM-dd'T'HH:mm:ss.SSSSSS"), 'yyyy-MM-dd HH:mm:ss')
substring_index(start_time, 'T', -1)s_tm,
substring_index(end_time, 'T', -1)e_tm
start_date 2018-06-19 13:59:41
end_date 2018-06-19 14:01:17
desired output
01:36
Solution for Hive.
Difference in seconds:
select UNIX_TIMESTAMP('2018-06-19T14:01:17.000000',"yyyy-MM-dd'T'HH:mm:ss.SSSSSS")-
UNIX_TIMESTAMP('2018-06-19T13:59:41.000000',"yyyy-MM-dd'T'HH:mm:ss.SSSSSS") as seconds_diff
Result:
96
Now calculate difference in HH:mm:ss:
select concat_ws(':',lpad(floor(seconds_diff/3600),2,'0'), --HH
lpad(floor(seconds_diff%3600/60),2,'0'), --mm
lpad(floor(seconds_diff%3600%60),2,'0') --ss
)
from
(
select --calculate seconds difference
UNIX_TIMESTAMP('2018-06-19T14:01:17.000000',"yyyy-MM-dd'T'HH:mm:ss.SSSSSS")-
UNIX_TIMESTAMP('2018-06-19T13:59:41.000000',"yyyy-MM-dd'T'HH:mm:ss.SSSSSS") as seconds_diff
) s
Result:
OK
00:01:36
Time taken: 1.071 seconds, Fetched: 1 row(s)
See also this answer about format convertion: https://stackoverflow.com/a/23520257/2700344
The data in the table as below.
The column jobdate data type is string.
jobdate
1536945012211.kc
1536945014231.kc
1536945312809.kc
I want to convert it to time stamp as the format 2018-12-205 06:15:10.505
I have tried the following queries but returning NULL.
select jobdate,from_unixtime(unix_timestamp(substr(jobdate,1,14),'YYYY-MM-DD HH:mm:ss.SSS')) from job_log;
select jobdate,from_unixtime(unix_timestamp(jobdate,'YYYY-MM-DD HH:mm:ss.SSS')) from job_log;
select jobdate,cast(date_format(jobdate,'YYYY-MM-DD HH:mm:ss.SSS') as timestamp) from job_log;
Please help me.
Thanks in advance
Original timestamps are too long, use 10 digits:
hive> select from_unixtime(cast(substr('1536945012211.kc',1,10) as int),'yyyy-MM-DD HH:mm:ss.SSS');
OK
2018-09-257 10:10:12.000
Time taken: 0.832 seconds, Fetched: 1 row(s)
hive> select from_unixtime(cast(substr('1536945012211.kc',1,10) as int),'yyyy-MM-dd HH:mm:ss.SSS');
OK
2018-09-14 10:10:12.000
Time taken: 0.061 seconds, Fetched: 1 row(s)
hive>
I have date in table as Sep 1 2017 2:00 PM as actualshipdate I want to convert it as 01-09-2017 in hive I try with below command but is showing null select actualshipdate,from_unixtime(unix_timestamp(substr(actualshipdate,0,11), 'dd-mm-yyyy')) as newdate from tablename;
Use unix_timestamp(string date, string pattern) to convert given date format to seconds passed from 1970-01-01. Then use from_unixtime() to convert to given format:
hive> select from_unixtime(unix_timestamp('Sep 1 2017 2:00 PM' ,'MMM dd yyyy HH:mm a'), 'dd-MM-yyyy');
OK
01-09-2017
Time taken: 0.049 seconds, Fetched: 1 row(s)
See patterns examples here: https://docs.oracle.com/javase/7/docs/api/java/text/SimpleDateFormat.html
You are using correct function but the parameters are wrong . The parameters should be unix_timestamp(datestring, date_format) ; this function will convert date to unix date format which you can format further by using from_unixtime(unixdateformat,format_youneed);
hive> select unix_timestamp('Sep 1 2017 2:00 PM' ,'MMM dd yyyy HH:mm a');
OK
1504256400
You need specific pattern of date for this you can use function
from_unixtime(unixdateformat,format_youneed);
hive> select from_unixtime(1504256400,'dd-MM-yyyy');
OK
01-09-2017
hive> select from_unixtime(UNIX_TIMESTAMP('Sep 1 2017 2:00 PM','MMM dd yyyy
HH:mm a'), 'dd-MM-yyyy');
OK
01-09-2017
Time taken: 0.135 seconds, Fetched: 1 row(s)
As you have date stored in actualdate in the table you can use below command to get the result.
**hive> select from_unixtime(UNIX_TIMESTAMP(actualshipdate,'MMM dd yyyy HH:mm a'), 'dd-MM-yyyy') from tablename;**
In am trying to automatically account for timezone in my postgres query. My goal is to select records that are between 0:00 and 23:59 in EST or UTC - 5:00.
The current query only returns records between 0:00 and 23:59 in UTC time.
select path, start_time from routes where routes.network_id = 1 and routes.start_time between '2017-06-13 00:00:00'::timestamp AND '2017-06-13 23:59:59'::timestamp
the column start_time is a timestamp without timezone, so by default it is in UTC
SELECT pg_typeof("start_time") from routes limit 1;
returns timestamp without timezone
how would one write a query to account for 5 hours difference and convert start_time to UTC - 5?
Try this:
select path, start_time - interval '5 hours' as start_time_est
from routes
where routes.network_id = 1
and routes.start_time between '2017-06-13 00:00:00-5'::timestamp with time zone
AND '2017-06-13 23:59:59-5'::timestamp with time zone;
I have a timestamp value like "Nov 27, 2016 8:30:00 AM" which I want to convert TIMESTAMP(6) format i.e. YYYY-MM-DD HH:MM:SS.fffffffff
Can anyone suggest how easily can this be achieved in hiveQL.
Use the unix_timsestamp function to get the date
hive> Select unix_timestamp("NOV 27, 2017", "MMM DD, YYYY") from xyz;
OK
1483257600
Time taken: 0.082 seconds, Fetched: 1 row(s)
hive> Select from_unixtime(unix_timestamp("NOV 27,2016 8:30:00 AM", "MMM dd,yyyy HH:mm:ss aa")) from xyz;
OK
2016-11-27 08:30:00
Time taken: 0.075 seconds, Fetched: 1 row(s)
Other options for unix_timestamp can be found here