I have data in a BigQuery instance with a some date fields in epoch/timestamp format. I'm trying to convert to a YYYYMMDD format or similar in order to create a report in Data Studio. I have tried the following solutions so far:
Change the format in the Edit Connection menu when creating the Data Source in Data Studio to Date format. Not working. I get Configuration errors when I add the field to the Data Studio report.
Create a new field using the TODATE() function. I always get an invalid formula error (even when I follow the documentation for this function). I have tried to change the field type prior to use the TODATE() function. Not working in any case.
Am I doing something wrong? Why do I always get errors?
Thanks!
The function for TODATE() is actually CURRENT_DATE(). Change timestamp to DATE using EXTRACT(DATE from variableName)
make sure not use Legacy SQL !
The issue stayed, but changing the name of the variable from actual_delivery_date to ADelDate made it work. So I presume there's a bug and short(er) names may help to avoid it
As commented by Elliott Brossard, the solution would be instead of using Data Studio for the conversion,use PARSE_DATE or PARSE_TIMESTAMP in BigQuery and convert it there instead.
Related
I am attempting to load a database from a CSV file using AsterixDB. Currently, it works using only string, int, and double fields. However, I have a column in the CSV file that is in DateTime format. Currently I am importing them as strings, which works fine, but I would like to import them as the SQL DateTime data type. When I try changing my schema and reimporting I get the following error:
ERROR: Code: 1 "org.apache.hyracks.algebricks.common.exceptions.NotImplementedException: No value parser factory for fields of type datetime"
All entries are in this format 02/20/2010 12:00:00 AM.
I know this isn't exactly inline with the format specified by the Asterix Data Model, however, I tried a test line with the proper format and the error persisted.
Does this mean AsterixDB cant parse DateTime when doing mass imports? And if so how can I get around this issue?
Any help is much appreciated.
Alright, after discussing with some colleagues, we believe that AsterixDB does not currently support DateTime parsing when mass importing. Our solution was to upsert every entry in the dataset with the parsing built into the query.
We used the following query:
upsert into csv_set (
SELECT parse_datetime(c.Date_Rptd, "M/D/Y h:m:s a") as Datetime_Rptd,
parse_datetime(c.Date_OCC, "M/D/Y h:m:s a") as Datetime_OCC,
c.*
FROM csv_set c
);
As you can see we parse the strings using the parse_datetime function from the AsterixDB Temporal Functions library. This query intentionally doesn't erase the column with the DateTimes in string format, although that would be very simple to do if your application requires it. If anyone has a better or more elegant solution please feel free to add to this thread!
I wish to create a table using edit text. The reason that I don't use auto detect is that my date format is not in the US format and Bigquery doesn't understand it (I'm not sure if it is a bug or I'm missing something). When I use Auto detect, Bigquery reads the date as mm/dd/yyyy while my date format is dd/mm/yyy.
I've tried the below but I'm getting an error
State:string,
New_Existing:string,
PARSE_DATE('%d/%m/%Y', Prd_Dt):DATE
You can use JSON to write your schema using edit text, refer to this GCP Documentation. Using your scenario below this is how we applied it to JSON:
Output:
However, you can only use the PARSE_DATE function in Bigquery when doing DML statements such as SELECT as per this GCP Documentation.
The latest version of Tableau has started using standard SQL when it connects to Google's BigQuery.
I recently tried to update a large table but found that there appeared to be errors when trying to parse datetimes. The table originates as a CSV which is loaded into BigQuery where further manipulations happen. The datetime column in the original CSV contain strings in ISO standard date time format (basically yyyy-mm-dd hh:mm). This saves a lot of annoying manipulation later.
But on trying to convert the datetime strings in Tableau into dates or datetimes I got a bunch of errors. On investigation they seemed to come from BigQuery and looked like this:
Error: Invalid timestamp: '2015-06-28 02:01'
I thought at first this might be a Tableau issue so I loaded a chunk of the original CSV into Tableau directly where the conversion of the string to a data worked perfectly well.
I then tried simpler versions of the conversion (to a year rather than a full datetime) and they still failed. The generated SQL for the simplest conversion looks like this:
SELECT
EXTRACT(YEAR
FROM
CAST(`Arrival_Date` AS TIMESTAMP)) AS `yr_Arrival_Date_ok`
FROM
`some_dataset`.`some_table` `some_table`
GROUP BY
1
The invalid timestamp in the error message always looks to me like a perfectly valid timestamp. And further analysis suggests it doesn't happen for all the rows in the source table, just occasional ones.
This error did not appear in older versions of Tableau/BigQuery where legacy SQL was the default for Tableau. So i'm presuming it is a consequence of standard SQL.
So is there an intermittent problem with casting to timestamps in BigQuery? Or is this a Tableau problem which causes the SQL to be incorrectly formatted? And what can I do about it?
The seconds part in the canonical timestamp representation required if the hour and minute are also present. Try this instead with PARSE_TIMESTAMP and see if it works:
SELECT
EXTRACT(YEAR
FROM
PARSE_TIMESTAMP('%F %R', `Arrival_Date`)) AS `yr_Arrival_Date_ok`
FROM
`some_dataset`.`some_table`.`some_table`
GROUP BY
1
How do you convert SQL mm/dd/yy datetime to mm/dd only? On Microsoft server.
Thanks all.
With dates and times it is an extremely common mistake to believe that what you see is what is stored. If the field is date, datetime, smalldatetime or datetime2 then what is stored are integers, not strings. So if the field is one of these, then:
convert(varchar(5),[date_field],1)
or
format([date_field],'MM/dd') -- mssql 2012 onward
If the information is a string already then left() will do the job.
Since you have specified an input format, the input must already be a string. Simply truncate with
cast(dateIn as char(5)).
You can use LEFT to just return the day and month:
SELECT LEFT('12/12/2000', 5)
I realize this isn't directly answering your question the way you asked it, but the best advice I can give is: Don't.
Instead, send back the field in its native datetime type. The database is not the place to be doing formatting. Instead, format the date in your application code.
For example, if you are calling SQL Server from a C#/.NET application, you could retrieve the value from a DataReader like this:
DateTime dt = (DateTime) reader["YourDateTime"];
Then you would format it as a string like this:
string s = dt.ToString("MM/dd");
This will ensure that the date is formatted correctly. If you are using a different language to call SQL Server, there are probably similar methods in that language.
One of the problems with the other approach mentioned (trunacating the string) is that the original value might not be formatted in mm/dd/yyyy to begin with. That all depends on the environment settings where the SQL Server is running. If you run the same code on an environment with dd/mm/yyyy settings, you would have unexpected results. This is avoided by using the native data type, the way I described.
I need to update a FileMaker Timestamp field with a timestamp taken from PHP and put into a script using the PHP API and executeSQL API and plugin
so
UPDATE table SET time ='2011-05-27 11:28:57'
My Question is as follows, how do I utilise the available scripting functions within Filemaker Pro 11 to convert the string that is being supplied within the SQL statement to an acceptable TimeStamp format for FileMake? or is it possible using the executeSQL plugin for FileMaker to do the conversion within the ExecuteSQL() function within the Execute SQL plugin?
I haven't tried it out, but it should work using CAST:
CAST( expression AS type [ (length) ] )
so, it should read:
UPDATE table SET time = CAST ('2011-05-27 11:28:57' AS TIMESTAMP)
However, please be aware that Filemaker's own ExecuteSQL() functions doesn't support UPDATE or INSERT INTO statements. You need to get a free extension from Dracoventions called epSQLExecute() in order to do this.
Hope this helps (someone).
Gary
You haven't given us much to go on, but my guess would be that you are updating a timestamp column with a string that does not match the required format.
You should convert your string to the appropriate object and then the update should work.