I want to md5 a timestamp column in Hive, without the millisecond.
If timestamp is before Epoch Unix Time (year 1970), timestamp is corrupted:
START_DATE=1915-07-15 23:25:26.290448384
select ID, START_DATE, MD5(START_DATE) from TABLE1
Result : START_DATE = 2500-02-02 00:00:00.0
No issue without adding MD5 function, or if the timestamp > 1970.
I've tried with vectorized parameter (https://cwiki.apache.org/confluence/display/hive/vectorized+query+execution#VectorizedQueryExecution-Limitations) but still the same issue.
Also tried : Cast as string, substr... before MD5.
How can we handle timestamp < 1970 ?
can you use somethign like this ?
select md5(from_unixtime(unix_timestamp(substring('1915-07-15 23:25:26.290448384',1,19)))) as md5_out
Unixtime will calculate any date before 1970 as negative number and calculate accordingly. so i think it should be alright.
Related
I have a Oracle SQL statement where I have to get the current timestamp as one of the columns. But I dont require the Timezone which CURRENT_TIMESTAMP gives or the AM/PM given by LOCALTIMESTAMP.
I require the current timestamp in 24hr format without the timezone.
Is it possible to get that in Oracle SQL?
It seems you're mixing 2 concepts here: "datatype" and "date format mask".
data type: LOCALTIMESTAMP returns datatype TIMESTAMP and CURRENT_TIMESTAMP returns datatype TIMESTAMP WITH TIME ZONE. TIMESTAMP is similar to DATE but has a higher precision. As usual... checking the docs is worth it.
date format mask: determines how you display the date information. Americans can't read 24 hour format, the rest of the world is confused by AM/PM. Fortunately, you can decide how you want to display the date as explained in the oracle docs.
If you just want to return the current date in 24 hour format you could do something like:
SELECT
TO_CHAR(SYSDATE,'DD-MON-YYYY HH24:MI:SS') as mydate,
<other columns>
FROM
<table_name>
If you need the date to be more precise and you require fractional seconds then you can use SYSTIMESTAMP instead of DATE with a format mask 'DD-MON-YYYY HH24:MI:SS.FF9'
I have a temp table.
It has last_update column in 2/10/2018 6:01:50 PM datetime format.
How can I write THE BEST QUERY to display all information that's updated on 02-Oct-2018 day?
You can use trunc function
select *
from tab
where trunc(last_update) = date'2018-10-02'
It is preferable to avoid TRUNC especially if you have an index on the column last_update.
A simple where condition should be better and may be better performant.
WHERE last_update >= date '2018-10-02' AND
last_update < date '2018-10-02' + 1
Use trunc function for getting the same day:
trunc(last_update) = trunc(to_date('02-Oct-2018', 'DD-MONTH-YYYY'))
The TRUNC (date) function returns date with the time portion of the day truncated to the unit specified by the format model fmt. The value returned is always of datatype DATE, even if you specify a different datetime datatype for date. If you omit fmt, then date is truncated to the nearest day.
You can also use format DD-MON-YYYY
I have a column eventtime that only stores the time of day as string. Eg:
0445AM - means 04:45 AM. I am using the below query to convert to UNIX timestamp.
select unix_timestamp(eventtime,'hhmmaa'),eventtime from data_raw limit 10;
This seems to work fine for test data. I always thought unixtimestamp is a combination of date and time while here I only have the time. My question is what date does it consider while executing the above function? The timestamps seem to be quite small.
Unix timestamp is the bigint number of seconds from Unix epoch (1970-01-01 00:00:00 UTC). The unix time stamp is a way to track time as a running total of seconds.
select unix_timestamp('0445AM','hhmmaa') as unixtimestamp
Returns
17100
And this is exactly 4hrs, 45min converted to seconds.
select 4*60*60 + 45*60
returns 17100
And to convert it back use from_unixtime function
select from_unixtime (17100,'hhmmaa')
returns:
0445AM
If you convert using format including date, you will see it assumes the date is 1970-01-01
select from_unixtime (17100,'yyyy-MM-dd hhmmaa')
returns:
1970-01-01 0445AM
See Hive functions dosc here.
Also there is very useful site about Unix timestamp
I have a value in my csv file for timetamp as '1522865628160'. When I load the data in bigQuery where this field type is timestamp, it saves the timestamp as '1522865628160000'. so when I query like
select * from <tablename> limit 1
it gives me error
Cannot return an invalid timestamp value of 1522865628160000000 microseconds relative to the Unix epoch. The range of valid timestamp values is [0001-01-1 00:00:00, 9999-12-31 23:59:59.999999]; error in writing field timestamp"
please help
I think the issue here is that you tried to load your UNIX timestamp data into a timestamp column in BigQuery. A BigQuery timestamp column is not the same thing as a UNIX timestamp. The latter is just a numerical value representing the number of seconds since the start of the UNIX epoch in 1970.
So the fix here would be to load your data into an INT64 (or INTEGER if you are using legacy) column. From there, you may convert your UNIX timestamp to a bona fide date or timestamp.
There is a MSEC_TO_TIMESTAMP() function which can convert an integer number of milliseconds since the UNIX epoch to a bona fide timestamp, e.g.
SELECT MSEC_TO_TIMESTAMP(1522865628160)
2018-04-04 11:13:48 UTC
I am storing a timestamp field in a SQLite3 column as TIMESTAMP DATETIME DEFAULT CURRENT_TIMESTAMP and I was wondering if there was any way for it to include milliseconds in the timestamp as well?
Instead of CURRENT_TIMESTAMP, use (STRFTIME('%Y-%m-%d %H:%M:%f', 'NOW')) so that your column definition become:
TIMESTAMP DATETIME DEFAULT(STRFTIME('%Y-%m-%d %H:%M:%f', 'NOW'))
For example:
CREATE TABLE IF NOT EXISTS event
(when_ts DATETIME DEFAULT(STRFTIME('%Y-%m-%d %H:%M:%f', 'NOW')));
To get number of milliseconds since epoch you can use julianday() with some additional calculations:
-- Julian time to Epoch MS
SELECT CAST((julianday('now') - 2440587.5)*86400000 AS INTEGER);
The following method doesn't require any multiplies or divides and should always produce the correct result, as multiple calls to get 'now' in a single query should always return the same result:
SELECT strftime('%s','now') || substr(strftime('%f','now'),4);
The generates the number of seconds and concatenates it to the milliseconds part from the current second+millisecond.
Here's a query that will generate a timestamp as a string with milliseconds:
select strftime("%Y-%m-%d %H:%M:%f", "now");
If you're really bent on using a numeric representation, you could use:
select julianday("now");
The accepted answer only gives you UTC. If you need a local time instead of UTC, use this:
strftime('%Y-%m-%d %H:%M:%f', 'now', 'localtime')