BigQuery TIMESTAMP datatype has microsecond precision, 6 fractional seconds.
When I run the following query
SELECT CAST("2020-06-02 07:00:53.001000" AS TIMESTAMP) AS as_timestamp
I would expect 2020-06-02 07:00:53.001000 UTC
What I get instead is ... 2020-06-02 07:00:53.1000 UTC
As there is 2 leadings 0's, BigQuery omits them for some reason. Can anyone help me out at all to stop BigQuery omitting these leadings 0s ? I'm trying to calculate some time differences between timestamps and it's throwing my calculations off.
Thanks
I strongly believe this is a UI bug, not a BigQuery Engine's
Below two proves for this
Prove 1
Look at JSON tab to see actual value returned by BQ
Prove 2
I run same query in another BigQuery IDE ( I personally use Goliath BigQuery IDE) and you can see correct result
Related
I am very new to data analytics and I need some help troubleshooting a SQL error I got. So, I have a column in this table which transferred over from Excel to SQL as a string type rather than a time piece of data. I want to make it into a time type so i can further analyze it.
So, I did the attached query to try and change the type of data using the CAST function. . However, it could not complete the query thanks to an outlier in the data set I have yet to clean the data and this was one of my first steps to so, but how do I remove this particular row that contains the invalid time string so the query can actually work? Or is there a better way to convert this entire column from text string to time?
BigQuery Time types adjust values outside the 24 hour boundary - 00:00:00 to 24:00:00; for example, if you subtract an hour from 00:30:00, the returned value is 23:30:00.
Based on your screenshot it looks like you are storing a duration? So 330 hours, 25 minutes and 55 seconds?
You would probably be best using timestamp, converting the hours to days and adding the remainder to your minutes and seconds.
You can then cast the resulting string to timestamp.
Edit
A much simpler solution is just cast('330:25:55' as interval) - thanks to #MatBailie
We are extracting data from a third-party database for export. Two of the columns are Timestamp columns with some of the values displayed below. The timestamp is supposed to represent a UTC timestamp from a GPS device. They are stored as int data types in an SQL Server database.
Any idea how I can convert this timestamp (e.g. 368815303) to a regular date/time? The numbers seen should be very recent - i.e. within Sept 2020 and should represent the time down to the nearest second.
Based on the comment, you would use:
select dateadd(second, 368815303, '1980-01-06')
Based on your expectation, the base time appears to be about 2009-01-01, which suggests:
select dateadd(second, 368815303, '2009-01-01')
I am not familiar with any date/time epoch that uses that as the base time. It might be some bespoke system.
It looks like some kind of epoch Unix timestamp.
Try this Epoch Converter which converts timestamp to human date
My solution is the following
SELECT TIMESTAMP_ADD('1970-01-01', INTERVAL 1551692341 SECOND) AS ts
Is there any other, more readable, way to convert a unix timestamp to a datetime ?
Yes there is.
TIMESTAMP_SECONDS(int64_expression). Description. Interprets
int64_expression as the number of seconds since 1970-01-01 00:00:00
UTC
Example:
SELECT timestamp_seconds(1551692341)
returns
2019-03-04 09:39:01 UTC
I am curious, why would you want anything simpler than what you already have.
Alternatively, BQ has support for UDF (user defined function) wherein you can create TEMP functions embedded with your JS snippet to parse the epoch value and convert into a required date value in any format deemed fit for data.
In most cases, it is good to have all such formatting and value transformations at the app layer and not create bespoke utilities at DB layer.
In any case, you may want to have a quick read at here
How to find the last DML or DQL update timestamp for Hive table. I can find TransientDDLid by using "formatted describe ". But it is helping in getting Modified Date. How can I figure out the latest UPDATED DATE for a Hive Table(Managed/External)?
Do show table extended like 'table_name';
It will give number of milliseconds elapsed since epoch.
Copy that number, remove last 3 digits and do select from_unixtime(no. of milliseconds elapsed since epoch)
e.g. select from_unixtime(1532442615733);
This will give you timestamp of that moment in current system's time zone.
I guess this is what you're looking for...
A previous solution regarding obtaining an answer in milliseconds for differences between two timestamps does not work in Hive 1.0 on Amazon EMR. Hive returns a blank column when casting a timestamp as double in my testing today. No errors are thrown when doing the CAST. Being able to calculate a time difference in fractions of a second between two columns of type "timestamp" are critical to our analysis. Any ideas?
You should try to convert into unix_timestamp using unix_timestamp(timestamp) but I think you will still be losing milliseconds.
select (unix_timestamp(DATE1)-unix_timestamp(DATE2)) TIMEDIFF from TABLE;