I am trying select a date from an Oracle DB (1 row result) and assign its value to an SSIS variable. I tried to convert it to_char (which would also work great to me) but I still get an error "SSIS value does not fall within the expected range"
You'll need to ensure your SSIS variable is of type String.
I expect you'll still need to convert the Oracle date to a string data type with the to_char method but I would have to test and don't have an Oracle instance I can test against.
Apparently the problem was about the name of the Result Name from whatever I was using to '1'. The helpful post that solve my problem is: https://dba.stackexchange.com/questions/129106/ssis-odbc-mapping-result-set-columns-to-variables-returns-error-value-does-not
Related
I'm trying to query a BigQuery table that has column "date" (set to type DATE in the schema) formatted as yyyy-mm-dd-??. In other words, there's an extra set of information about the date and I'm not really sure what it is. When I try to query the "date" column I run into the error:
SQL Error [100032] [HY000]: [Simba]BigQueryJDBCDriver Error executing query job. Message: Invalid date: '2022-09-03-01'
I've tried cast(date as string), cast(left(date, 10) as string), all types of workarounds, but the error persists. It seems that no matter how much I try and nail it home in the query that I want this weird date column to be read as a string, so that I can work with it, BigQuery still wants to take it as a date, I guess because that's how it's setup in the schema. I don't care if this is parsed into a date properly or if it's read as a string and then I can parse it from there, I just want to be able to query the date column without getting an error.
I use a Azure Datafactory Pipeline.
Within that pipeline i use 2 activities:
Lookup to get a date value
This is the output:
"firstRow": {
"Date": "2022-10-26T00:00:00Z"
A dataflow which is getting the date from the lookup in 1 which is used in the source options SQL query in the where clause:
This is the query:
"SELECT ProductID ,ProductName ,SupplierID,CategoryID ,QuantityPerUnit ,UnitPrice ,UnitsInStock,UnitsOnOrder,ReorderLevel,Discontinued,LastModifiedDate FROM Noordwind.Products where LastModifiedDate >= '{$DS_LastPipeLineRunDate}'"
When i fill the parameter by hand with for example '2022-10-26' then it works great, but when i let the parameter get's its value from the Lookup in step 1 the dataflow fails
Error message:
{"message":"Job failed due to reason: Converting to a date or time failed due to an invalid character. Details:null","failureType":"UserError","target":"Products","errorCode":"DF-Executor-Conversion"}
This is the parameter in the pipeline view, but clicked on the dataflow:
I have tried casting the date al kind of things but not the right thing.
Can you help me.
UPDATE:
After a question from Rakesh:
This is the activity parameter
#activity('LookupLastPipelineRunDate').output.firstRow
I have reproduced the above and got the below results.
My source sample data from SQL database.
For demo, I have used set variable for the date and given a sample date like below.
Created a string parameter and given this variable value to it.
In your case pass the lookup firstrow output date.
I have used below dataflow expression in the query of dataflow source and got the desired result.
concat('select * from dbo.table1 where d1 >=','\'',$date_value,'\'')
Result in a target SQL table.
I have created an activity set variable:
The first pipeline still returns the right date.
I even converted it just to be sure to datetime.
I can create a variable with type string.
Code:
#activity('LookupLastPipelineRunDate').output.firstRow
Regardless of the activity set variable that fails, it looks like the date enters nicely as an input in the Set variable activity
And still a get an error:
When i read this error message, it says that you can't put a date in a string variable. But i can only choose string, boolean and array, so there is no better option for this.
I also reviewd this website.
enter link description here
There for i have altered the table which contains the source data which i use in the dataflow.
I Deleted the column LastModifiedDate because it has datatype datetime.
Now i created the same column with datatype datetime2
I did this because i read that datetime2 has less problems with conversions.
Inspired by the slected answer to Declare a variable in RedShift I am trying to use a query result as the format value in a to_char function call:
WITH tmp_variables as (
select 'YYYY-MM-DD' as date_format
)
SELECT to_char(OrderDate, (SELECT date_format FROM tmp_variables)) FROM Orders
But I am getting an error
TO_CHAR parameter: Second input must be a string literal
How can the tmp_variables's date_format value be used as a to_char format without getting an error or is there an alternative to using to_char where this would work?
SELECT is a SQL operator that work upon data. SQL is compiled before it can operate on data. The basic answer is that this won't work as written.
What you are trying to achieve isn't clear in the question - change date output format for some reason for some set of queries but not others? In the general case you will need to modify the SQL that goes to the compiler which will mean reading some configuration and merging this into the SQL text. If the use case is more limited there may be another way to achieve the desired result but only within some set of limitations.
Some possibilities - You could set a SQL variable with the format literal. Your client can read info and modify the query itself if it is capabile. A stored procedure could be used. A SQL modifier (pg_bouncer?) could live between the client and the cluster and substitute the string based on some other factors. Each of these has limitations and costs.
If you can describe the use case it could generate different / better ways.
I have data in a BigQuery instance with a some date fields in epoch/timestamp format. I'm trying to convert to a YYYYMMDD format or similar in order to create a report in Data Studio. I have tried the following solutions so far:
Change the format in the Edit Connection menu when creating the Data Source in Data Studio to Date format. Not working. I get Configuration errors when I add the field to the Data Studio report.
Create a new field using the TODATE() function. I always get an invalid formula error (even when I follow the documentation for this function). I have tried to change the field type prior to use the TODATE() function. Not working in any case.
Am I doing something wrong? Why do I always get errors?
Thanks!
The function for TODATE() is actually CURRENT_DATE(). Change timestamp to DATE using EXTRACT(DATE from variableName)
make sure not use Legacy SQL !
The issue stayed, but changing the name of the variable from actual_delivery_date to ADelDate made it work. So I presume there's a bug and short(er) names may help to avoid it
As commented by Elliott Brossard, the solution would be instead of using Data Studio for the conversion,use PARSE_DATE or PARSE_TIMESTAMP in BigQuery and convert it there instead.
How do you convert SQL mm/dd/yy datetime to mm/dd only? On Microsoft server.
Thanks all.
With dates and times it is an extremely common mistake to believe that what you see is what is stored. If the field is date, datetime, smalldatetime or datetime2 then what is stored are integers, not strings. So if the field is one of these, then:
convert(varchar(5),[date_field],1)
or
format([date_field],'MM/dd') -- mssql 2012 onward
If the information is a string already then left() will do the job.
Since you have specified an input format, the input must already be a string. Simply truncate with
cast(dateIn as char(5)).
You can use LEFT to just return the day and month:
SELECT LEFT('12/12/2000', 5)
I realize this isn't directly answering your question the way you asked it, but the best advice I can give is: Don't.
Instead, send back the field in its native datetime type. The database is not the place to be doing formatting. Instead, format the date in your application code.
For example, if you are calling SQL Server from a C#/.NET application, you could retrieve the value from a DataReader like this:
DateTime dt = (DateTime) reader["YourDateTime"];
Then you would format it as a string like this:
string s = dt.ToString("MM/dd");
This will ensure that the date is formatted correctly. If you are using a different language to call SQL Server, there are probably similar methods in that language.
One of the problems with the other approach mentioned (trunacating the string) is that the original value might not be formatted in mm/dd/yyyy to begin with. That all depends on the environment settings where the SQL Server is running. If you run the same code on an environment with dd/mm/yyyy settings, you would have unexpected results. This is avoided by using the native data type, the way I described.