I have two timestamp request and response. I need to find out difference between two these two timestamp in millisecond as below.
Request: 2020-03-20 10:00:00:010
Response: 2020-03-20 10:00:00:020
Diff: 10 millisecond
I tried but could not get my required answer.
I tried as below but it is giving me 0 instead of 10.
select (unix_timestamp(2020-03-20 10:00:00:010) - unix_timestamp(2020-03-20 10:00:00:020))
Thats because unix_timestamp trims out the millisecond portion.
You need some regex to parse it - something like:
select cast(regexp_replace('2020-03-20 10:00:00:020',
'(\\d{4})-(\\d{2})-(\\d{2}) (\\d{2}):(\\d{2}):(\\d{2}):(\\d{3})',
'$1-$2-$3 $4:$5:$6.$7') as timestamp);
OR
SELECT ROUND((CAST(CAST('2020-03-20 10:00:00.020' AS TIMESTAMP) AS DOUBLE)
- CAST(CAST('2020-03-20 10:00:00.010' AS TIMESTAMP) AS DOUBLE)) * 1000)
as timediff
The millisecond portion should of the form yyyy-mm-dd hh:mm:ss.SSS
So you may have to replace the ":" with "." for milliseconds.
Hive timestamp will be always something like:
2020-03-20 01:50:19.158
To get the difference between the two timestamps, you can try running below query:
select date_format("2020-03-20 10:00:00.020",'S') -date_format("2020-03-20 10:00:00.010",'S');
If the millisecond part is separated by ":" , then you can get the difference by running below query:
select cast(substr("2020-03-20 10:00:00:020",-3) as int) - cast(substr("2020-03-20 10:00:00:010",-3) as int);
Related
I am trying to cast dates from string format to numeric format in milliseconds trying to keep
also the .SSS part as I need to process data at the level of milliseconds duration. While in Hive 1.1.0 I am able to do that with the code below in the newer version it does not let me do that:
select current_timestamp(), unix_timestamp(current_timestamp(), 'yyyy-MM-dd HH:mm:ss.SSS')*1000, cast((cast(date_format(cast(current_timestamp() as string),'yyyy-MM-dd HH:mm:ss.SSS') as timestamp)) as double) * 1000 as time_milliseconds
Can you tell me a workaround to this?
Thank you
Extract millisecond part from string and add to the (timestamp in seconds)*1000
select current_timestamp(),
--unix_timestamp returns seconds only
unix_timestamp(current_timestamp())*1000, --without .SSS * 1000
unix_timestamp(current_timestamp())*1000 +
bigint(regexp_extract(string(current_timestamp()),'\\.(\\d+)$',1)) --with .SSS
Result:
2021-09-21 13:52:32.034 1632232352000 1632232352034
Explicit conversion to bigint and string may be not necessary.
One more method how you can get milliseconds part is to split string by dot and get element #1: split(current_timestamp(),'\\.')[1] instead of regexp_extract(string(current_timestamp()),'\\.(\\d+)$',1):
select ts, unix_timestamp(ts_splitted[0])*1000, unix_timestamp(ts_splitted[0]) * 1000 + ts_splitted[1]
from
(
select current_timestamp() ts, split(current_timestamp(),'\\.') ts_splitted
)s
Result:
2021-09-21 18:21:11.032 1632248471000 1632248471032
I prefer this method. Of course if you have timestamps with microseconds or nanoseconds, the logic should be adjusted accordingly based on the length of the fractional part.
I am trying to calculate the difference between two dates in an oracle database using a JDBC connection. I followed the advice from this question using a query like this:
SELECT CREATE_DATE - CLOSED
FROM TRANSACTIONS;
and I get the following error:
Incompatible value type specified for
column:CREATE_DATE-CLOSED. Column Type = 11 and Value Type =
8.[10176] Error Code: 10176
What should I change so I can successfully calculate the difference between the dates?
note: CREATE_DATE and CLOSED both have TIMESTAMP type
The answer you found is related to date datatypes, but you are dealing with timestamps. While substracting two Oracle dates returns a number, substracting timestamps produces an interval datatype. This is probably not what you want, and, apparently, your driver does not properly handle this datatype.
For this use case one solution is to cast the timestamps to dates before substracting them:
select cast(create_date as date) - cast(closed as date) from transactions;
As it was mentioned, it seems that JDBC cannot work with the INTERVAL datatype. What about casting it with the EXTRACT function to the expected output as number? If you want number of seconds between those two timestamps, it would be:
SELECT EXTRACT(SECOND FROM (CREATE_DATE - CLOSED)) FROM TRANSACTIONS;
Here are list of options which might be used instead of SECOND:
https://docs.oracle.com/database/121/SQLRF/functions067.htm#SQLRF00639
When we subtract one date from another Oracle gives us the difference as a number: it's straightforward arithmetic. But when we subtract one timestamp from another - which is what you're doing - the result is an INTERVAL. Older versions of JDBC don't like the INTERVAL datatype (docs) .
Here are a couple of workarounds, depending on what you want to do with the result. The first is to calculate the number of seconds from the interval result. extract second from ... only gives us the numbers of seconds in the interval. This will be fine providing none of your intervals are more than fifty-nine seconds long. Longer intervals require us to extract minute, hour and even days. So that solution would be:
select t.*
, extract (day from (t.closed - t.create_date)) * 84600
+ extract (hour from (t.closed - t.create_date)) * 3600
+ extract (minute from (t.closed - t.create_date)) * 60
+ extract (second from (t.closed - t.create_date)) as no_of_secs
from transactions t
A second solution is to follow the advice in the JDBC mapping guide and turn the interval into a string:
select t.*
, cast ((t.closed - t.create_date) as varchar2(128 char)) as intrvl_str
from transactions t
The format of a string interval is verbose:INTERVAL'+000000001 04:40:59.710000'DAY(9)TO SECOND. This may not be useful in the Java side of the application. But with regex we can turn it into a string which can be converted into a Java 8 Duration object (docs) : PnDTnHnMn.nS.
select t.id
, regexp_replace(cast ((t.closed - t.create_date) as varchar2(128 char))
, 'INTERVAL''\+([0-9]+) ([0-9]{2}):([0-9]{2}):([0-9]{2})\.([0-9]+)''DAY\(9\)TO SECOND'
, 'P\1DT\2H\3M\4.\5S')
as duration
from transactions t
There is a demo on db<>fiddle
NOTE: I am running my query in Qubole's presto and sql command engine.
I am trying to convert my string timestamp to just date but none of the options are working out.
My string timestamp looks like 2017-03-29 10:32:28.0
and I want to have it like 2017-03-29
I have tried following queries to convert this string timestamp to retrieve date
1. select cast(created as date) from table1
Value cannot be cast to date: 2017-05-26 17:23:58.0
2. select cast(from_iso8601_timestamp(created) as date) from table1
Invalid format: "2014-12-19 06:06:36.0" is malformed at " 06:06:36.0"
3. select date(created) from table1
Value cannot be cast to date: 2012-10-24 13:50:00.0
How I can convert this timestamp to date in presto/sql?
As far as explained in the documentation, prestoDB seems to expect timestamps in a format '2001-08-22 03:04:05.321', and dates in a '2001-08-22'.
One solution would be to use a string function to extract the relevant part of the string before converting it. We know that the date part is located before the first space in the string, so.
If you need the date part as a string datatype:
split_part(created, ' ', 1)
If you need the date part as a date datatype:
cast(split_part(created, ' ', 1) as date)
You can try to use one of the following solutions:
SELECT
'2017-03-29 10:32:28.0' AS input_string,
DATE(date_parse('2017-03-29 10:32:28.0', '%Y-%m%-%d %H:%i:%s.%f')) AS solution_1,
DATE(try_cast('2017-03-29 10:32:28.0' as timestamp)) AS solution_2
Recently there is an issue from my side.
In the table in redshift I have 2 columns:
The visit_time, which mentions the start time of the session and in YYYY-MM-DD HH:MM:SS format; Another column which is the time_spent, and it stands for the millesecond that user spends in certain page and currently it is in float(8)
What I want to do is to Add the visit_time with time_spent in second and convert it into YYYY-MM-DD HH:MM:SS, for example 2018-09-20 21:00:55 + 2 = 2018-09-20 21:00:57, so that I can get the visit_end_time. I tried to add it with date add function like this
Select
dateadd(SECOND,CAST (a.timeonpage AS DECIMAL)/1000 ,a.visit_time::date) time_left,
But it returns error: Invalid operation: function pg_catalog.date_add("unknown", double precision, date) does not exist
To do this, I tried to used the dateadd function like this:
Select
dateadd(SECOND,CAST (a.timeonpage AS INTEGER)/1000 ,a.visit_time::date) time_left,
It does not return error, but it returns the start of the date e.g 2018-09-23 00:00:00, which is not sth that I need.
What is the error that I made and how can I solve it?
Many thanks!
The dateadd function does require an integer for the interval. The problem with your second example is that when you cast a.visit_time to a date type it truncates it to the start of that day (removes the time component). Cast it to a timestamp instead:
select dateadd(second, round(a.timeonpage/1000.0)::integer, a.visit_time::timestamp);
select dateadd(second, round(2345/1000.0)::integer, '2018-09-20 21:00:55'::timestamp);
date_add
---------------------
2018-09-20 21:00:57
Also if you are storing visit_time as a string I would recommend making the column type a timestamp.
I'm working with Hive and I have a table structured as follows:
CREATE TABLE t1 (
id INT,
created TIMESTAMP,
some_value BIGINT
);
I need to find every row in t1 that is less than 180 days old. The following query yields no rows even though there is data present in the table that matches the search predicate.
select *
from t1
where created > date_sub(from_unixtime(unix_timestamp()), 180);
What is the appropriate way to perform a date comparison in Hive?
How about:
where unix_timestamp() - created < 180 * 24 * 60 * 60
Date math is usually simplest if you can just do it with the actual timestamp values.
Or do you want it to only cut off on whole days? Then I think the problem is with how you are converting back and forth between ints and strings. Try:
where created > unix_timestamp(date_sub(from_unixtime(unix_timestamp(),'yyyy-MM-dd'),180),'yyyy-MM-dd')
Walking through each UDF:
unix_timestamp() returns an int: current time in seconds since epoch
from_unixtime(,'yyyy-MM-dd') converts to a string of the given format, e.g. '2012-12-28'
date_sub(,180) subtracts 180 days from that string, and returns a new string in the same format.
unix_timestamp(,'yyyy-MM-dd') converts that string back to an int
If that's all getting too hairy, you can always write a UDF to do it yourself.
Alternatively you may also use datediff. Then the where clause would be
in case of String timestamp (jdbc format) :
datediff(from_unixtime(unix_timestamp()), created) < 180;
in case of Unix epoch time:
datediff(from_unixtime(unix_timestamp()), from_unixtime(created)) < 180;
I think maybe it's a Hive bug dealing with the timestamp type. I've been trying to use it recently and getting incorrect results.
If I change your schema to use a string instead of timestamp, and supply values in the
yyyy-MM-dd HH:mm:ss
format, then the select query worked for me.
According to the documentation, Hive should be able to convert a BIGINT representing epoch seconds to a timestamp, and that all existing datetime UDFs work with the timestamp data type.
with this simple query:
select from_unixtime(unix_timestamp()), cast(unix_timestamp() as
timestamp) from test_tt limit 1;
I would expect both fields to be the same, but I get:
2012-12-29 00:47:43 1970-01-16 16:52:22.063
I'm seeing other weirdness as well.
TIMESTAMP is milliseconds
unix_timestamp is in seconds
You need to multiply the RHS by 1000.
where created > 1000 * date_sub(from_unixtime(unix_timestamp()), 180);
After reviewing this and referring to Date Difference less than 15 minutes in Hive I came up with a solution. While I'm not sure why Hive doesn't perform the comparison effectively on dates as strings (they should sort and compare lexicographically), the following solution works:
FROM (
SELECT id, value,
unix_timestamp(created) c_ts,
unix_timestamp(date_sub(from_unixtime(unix_timestamp()), 180), 'yyyy-MM-dd') c180_ts
FROM t1
) x
JOIN t1 t ON x.id = t.id
SELECT to_date(t.Created),
x.id, AVG(COALESCE(x.HighestPrice, 0)), AVG(COALESCE(x.LowestPrice, 0))
WHERE unix_timestamp(t.Created) > x.c180_ts
GROUP BY to_date(t.Created), x.id ;