I'm trying to query a BigQuery table that has column "date" (set to type DATE in the schema) formatted as yyyy-mm-dd-??. In other words, there's an extra set of information about the date and I'm not really sure what it is. When I try to query the "date" column I run into the error:
SQL Error [100032] [HY000]: [Simba]BigQueryJDBCDriver Error executing query job. Message: Invalid date: '2022-09-03-01'
I've tried cast(date as string), cast(left(date, 10) as string), all types of workarounds, but the error persists. It seems that no matter how much I try and nail it home in the query that I want this weird date column to be read as a string, so that I can work with it, BigQuery still wants to take it as a date, I guess because that's how it's setup in the schema. I don't care if this is parsed into a date properly or if it's read as a string and then I can parse it from there, I just want to be able to query the date column without getting an error.
Related
I use a Azure Datafactory Pipeline.
Within that pipeline i use 2 activities:
Lookup to get a date value
This is the output:
"firstRow": {
"Date": "2022-10-26T00:00:00Z"
A dataflow which is getting the date from the lookup in 1 which is used in the source options SQL query in the where clause:
This is the query:
"SELECT ProductID ,ProductName ,SupplierID,CategoryID ,QuantityPerUnit ,UnitPrice ,UnitsInStock,UnitsOnOrder,ReorderLevel,Discontinued,LastModifiedDate FROM Noordwind.Products where LastModifiedDate >= '{$DS_LastPipeLineRunDate}'"
When i fill the parameter by hand with for example '2022-10-26' then it works great, but when i let the parameter get's its value from the Lookup in step 1 the dataflow fails
Error message:
{"message":"Job failed due to reason: Converting to a date or time failed due to an invalid character. Details:null","failureType":"UserError","target":"Products","errorCode":"DF-Executor-Conversion"}
This is the parameter in the pipeline view, but clicked on the dataflow:
I have tried casting the date al kind of things but not the right thing.
Can you help me.
UPDATE:
After a question from Rakesh:
This is the activity parameter
#activity('LookupLastPipelineRunDate').output.firstRow
I have reproduced the above and got the below results.
My source sample data from SQL database.
For demo, I have used set variable for the date and given a sample date like below.
Created a string parameter and given this variable value to it.
In your case pass the lookup firstrow output date.
I have used below dataflow expression in the query of dataflow source and got the desired result.
concat('select * from dbo.table1 where d1 >=','\'',$date_value,'\'')
Result in a target SQL table.
I have created an activity set variable:
The first pipeline still returns the right date.
I even converted it just to be sure to datetime.
I can create a variable with type string.
Code:
#activity('LookupLastPipelineRunDate').output.firstRow
Regardless of the activity set variable that fails, it looks like the date enters nicely as an input in the Set variable activity
And still a get an error:
When i read this error message, it says that you can't put a date in a string variable. But i can only choose string, boolean and array, so there is no better option for this.
I also reviewd this website.
enter link description here
There for i have altered the table which contains the source data which i use in the dataflow.
I Deleted the column LastModifiedDate because it has datatype datetime.
Now i created the same column with datatype datetime2
I did this because i read that datetime2 has less problems with conversions.
I am trying to extract the time from a datetime column in my Amazon Redshift database (Postgresql 8.0). I have already referred to previous questions such as this. But I am getting an unusual error.
When I try:
SELECT collected_timestamp::time
or
SELECT cast(collected_timestamp as time)
I get the following error:
ERROR: Specified types or functions (one per INFO message) not supported on Redshift tables
The goal is to pull the time portion from the timestamp such that 2017-11-06 13:03:28 returns 13:03:28.
This seems like an easy problem to solve but for some reason I am missing something. Researching that error does not lead to anything meaningful. Any help is appreciated.
Note that Redshift <> PostgreSQL - it was forked from PostgreSQL but is very different under the hood.
You're trying to cast a timestamp value to a data type of "time" which does not exist in Redshift. To return a value that is only the time component of a timestamp you will need to cast it to a character data type, e.g.:
SELECT to_char(collected_timestamp, 'HH24:MI:SS');
There are a few ways, here is one i use:
SELECT ('2018-03-07 21:55:12'::timestamp - trunc('2018-03-07 21:55:12'::timestamp))::time;
I hope that helps.
EDIT: I have made incorrect use of ::time please see comments on other answer.
I am trying select a date from an Oracle DB (1 row result) and assign its value to an SSIS variable. I tried to convert it to_char (which would also work great to me) but I still get an error "SSIS value does not fall within the expected range"
You'll need to ensure your SSIS variable is of type String.
I expect you'll still need to convert the Oracle date to a string data type with the to_char method but I would have to test and don't have an Oracle instance I can test against.
Apparently the problem was about the name of the Result Name from whatever I was using to '1'. The helpful post that solve my problem is: https://dba.stackexchange.com/questions/129106/ssis-odbc-mapping-result-set-columns-to-variables-returns-error-value-does-not
I am reading data from data lake (csv) and when running the below query, I am getting a 'Conversion failed when converting date and/or time from character string' error message.
select convert(datetime, NullIf(ltrim(rtrim([Date started])), ''), 111)
FROM dl.temp
Looked through the data and checked the source file as well, couldn't spot anything unusual.
As soon as I include the * and change the query to the below everything runs fine and the conversion seem to be doing its job.
select convert(datetime, NullIf(ltrim(rtrim([Date started])), ''), 111),*
from dl.temp
Out of curiosity also wanted to check the max and minimum date, so running max gives me the following:
However when I search for that particular value like below, I don't get any rows returned. It seems like it setting it to the column name. Does anyone know what is going on?
select *
from dl.temp
where [Date started] = 'Date started'
I am running this against an Azure Data Warehouse.
I think you'll find the issue is in your external file format.
In the CREATE EXTERNAL FILE FORMAT you probably need to add FIRST_ROW=2 in your FORMAT OPTIONS.
https://learn.microsoft.com/en-us/sql/t-sql/statements/create-external-file-format-transact-sql
The latest version of Tableau has started using standard SQL when it connects to Google's BigQuery.
I recently tried to update a large table but found that there appeared to be errors when trying to parse datetimes. The table originates as a CSV which is loaded into BigQuery where further manipulations happen. The datetime column in the original CSV contain strings in ISO standard date time format (basically yyyy-mm-dd hh:mm). This saves a lot of annoying manipulation later.
But on trying to convert the datetime strings in Tableau into dates or datetimes I got a bunch of errors. On investigation they seemed to come from BigQuery and looked like this:
Error: Invalid timestamp: '2015-06-28 02:01'
I thought at first this might be a Tableau issue so I loaded a chunk of the original CSV into Tableau directly where the conversion of the string to a data worked perfectly well.
I then tried simpler versions of the conversion (to a year rather than a full datetime) and they still failed. The generated SQL for the simplest conversion looks like this:
SELECT
EXTRACT(YEAR
FROM
CAST(`Arrival_Date` AS TIMESTAMP)) AS `yr_Arrival_Date_ok`
FROM
`some_dataset`.`some_table` `some_table`
GROUP BY
1
The invalid timestamp in the error message always looks to me like a perfectly valid timestamp. And further analysis suggests it doesn't happen for all the rows in the source table, just occasional ones.
This error did not appear in older versions of Tableau/BigQuery where legacy SQL was the default for Tableau. So i'm presuming it is a consequence of standard SQL.
So is there an intermittent problem with casting to timestamps in BigQuery? Or is this a Tableau problem which causes the SQL to be incorrectly formatted? And what can I do about it?
The seconds part in the canonical timestamp representation required if the hour and minute are also present. Try this instead with PARSE_TIMESTAMP and see if it works:
SELECT
EXTRACT(YEAR
FROM
PARSE_TIMESTAMP('%F %R', `Arrival_Date`)) AS `yr_Arrival_Date_ok`
FROM
`some_dataset`.`some_table`.`some_table`
GROUP BY
1