I am Passing the following Json input from Eventhub to Stream Analytics.
{"meter_totalcycleenergy":null,"Test2": 20}, {"meter_totalcycleenergy":40,"Test2":20}
But the job is failing stating the error.
Encountered error trying to write 1 event(s): Cannot convert from property 'meter_totalcycleenergy' of type 'System.String' to column 'meter_totalcycleenergy' of type 'System.Single'.
Error Image
How to handle such conditions.
I think Json nulls are not exactly SQL NULLs, so what would be the proper way to check for null values in a query?
Datatype of meter_totalcycleenergy is float in my database.
You can use is not null. For eg:
select *
from input
where meter_totalcycleenergy is not null
Related
I use a Azure Datafactory Pipeline.
Within that pipeline i use 2 activities:
Lookup to get a date value
This is the output:
"firstRow": {
"Date": "2022-10-26T00:00:00Z"
A dataflow which is getting the date from the lookup in 1 which is used in the source options SQL query in the where clause:
This is the query:
"SELECT ProductID ,ProductName ,SupplierID,CategoryID ,QuantityPerUnit ,UnitPrice ,UnitsInStock,UnitsOnOrder,ReorderLevel,Discontinued,LastModifiedDate FROM Noordwind.Products where LastModifiedDate >= '{$DS_LastPipeLineRunDate}'"
When i fill the parameter by hand with for example '2022-10-26' then it works great, but when i let the parameter get's its value from the Lookup in step 1 the dataflow fails
Error message:
{"message":"Job failed due to reason: Converting to a date or time failed due to an invalid character. Details:null","failureType":"UserError","target":"Products","errorCode":"DF-Executor-Conversion"}
This is the parameter in the pipeline view, but clicked on the dataflow:
I have tried casting the date al kind of things but not the right thing.
Can you help me.
UPDATE:
After a question from Rakesh:
This is the activity parameter
#activity('LookupLastPipelineRunDate').output.firstRow
I have reproduced the above and got the below results.
My source sample data from SQL database.
For demo, I have used set variable for the date and given a sample date like below.
Created a string parameter and given this variable value to it.
In your case pass the lookup firstrow output date.
I have used below dataflow expression in the query of dataflow source and got the desired result.
concat('select * from dbo.table1 where d1 >=','\'',$date_value,'\'')
Result in a target SQL table.
I have created an activity set variable:
The first pipeline still returns the right date.
I even converted it just to be sure to datetime.
I can create a variable with type string.
Code:
#activity('LookupLastPipelineRunDate').output.firstRow
Regardless of the activity set variable that fails, it looks like the date enters nicely as an input in the Set variable activity
And still a get an error:
When i read this error message, it says that you can't put a date in a string variable. But i can only choose string, boolean and array, so there is no better option for this.
I also reviewd this website.
enter link description here
There for i have altered the table which contains the source data which i use in the dataflow.
I Deleted the column LastModifiedDate because it has datatype datetime.
Now i created the same column with datatype datetime2
I did this because i read that datetime2 has less problems with conversions.
I have column in flatfile contain value like. 2021-12-15T02:40:39+01:00
When I tried to Insert to table whose column datatype is datetime2.
It throwing Error as :
The data conversion for column "Mycol" returned status value 2 and status text
"The value could not be converted because of a potential loss of data.".
What could be best datatype for such values.
It seems the problem is two-fold here. One, the destination column for your value should be a datetimeoffset(0) and two that SSIS doesn't support the format yyyy-MM-ddThh:mm:ss for a DT_DBTIMESTAMPOFFSET; the T causes it problems.
Therefore I suggest that you define the column, MyCol, in your Flat File Connection as a DT_STR. Them, in your data flow task, use a derived column transformation which replaces MyCol and uses the following expression to remove the T and with a space ( ):
(DT_DBTIMESTAMPOFFSET,0) (REPLACE(Mycol,"T"," "))
This will then cause the correct data type and value to be inserted into the database.
I have created spark SQL table like below through Azure Databricks:
create table sample1(price double)
Actual file has data like 'abc' instead of double value.
While inserting 'abc' string value into double column it accepts as NULL without any failure. My concern is why are we not getting any error? I want to failure message in this case.
Please let me know if I'm missing something. I want to disable the implicit conversion of datatypes.
My table consists of 253 columns so loaded data to pig bag without schema method, when I apply filter condition to any data getting this error how to avoid it? Any jars needs to be added ?
Error: IMPLICIT_CAST_TO_CHARARRAY
Since schema is not defined in the load statement, so default data type for all the fields are bytearray.
Pig tries to implicitly cast from one type to another while using in the script.
All castings are not possible, so in that case, warnings are thrown.
You can refer the Pig Cast operators for details
When the schema is not specified the fields are loaded as bytearray.When you apply filter you will have to cast the field you are using for the filter.If you are filtering using a string use (chararray) in front of the field.
I'm using the BI tool Domo, which uses Amazon Red Shift. I have a dataset that runs nightly using Zendesk data.
I'm getting this error:
OnboardFlowExecution(2794) data flow execution id
(724670342c4c48a9a61e7a617e6462c1) failed:
java.lang.NumberFormatException: For input string: "null"
I've researched the error and I am under the impression that somewhere in the data set a Number Column has data it doesn't like and is wreaking havoc with my downstream process.
How do I find the offending column/row ?
Amazon RedShift is expecting a string value in the column but receiving 'null' values instead. Hence, it's throwing an exception (a kind of error).
Add a transform to the input dataset to handle the null exception. We've previously used NVL function in RedShift to replace null values with something else.
E.g. Your transform could be:
Select employee_id, NVL(emp_first_name, 'No Name') from employees;
The NVL function will replace all the null values in the 'emp_first_name' column with 'No Name'