I have a situation where table has two main columns DataFieldName & DataFieldValue along with some identifier such as OrderNumber.
Now in DataFieldName there is value named as "OrderDate" and respective dates are coming in DataFieldValue.
But some of the value for "OrderDate" are coming as non date values. I need to validate such non date value based on the condition where DataFieldName has value as "OrderDate" then validate the DataFieldValue for valid date in SSIS.
You can split your data using a Conditional Split :
or a query with a condition if you are using SQL :
SELECT DataFieldName, DataFieldValue FROM yourTable
WHERE DataFieldName LIKE 'OrderDate'
If you are using SQL Server :
SELECT
CASE WHEN TRY_CONVERT(date, DataFieldValue) IS NULL
THEN 'Cast failed'
ELSE 'Cast succeeded'
END AS Result
FROM yourTable
If you are using Oracle :
SELECT cast(DataFieldValue AS NUMBER DEFAULT NULL ON CONVERSION ERROR) FROM yourTable
Or, you can use a Data Conversion Transformation component :
Then you can redirect the output to a flat file for example :
Related
I was executing below simple Spark-SQL code azure databricks.
val df2=spark.sql(
s"""
select
mbrgm.mbrgm_id as case_id,
case
when mbr_hist.meck is not null
and mbr_hist.efdt is not null
and mbr_hist.efdt <= mbr_pgm.credttm
and (
mbr_hist.exp_dt is null
or mbr_hist.exp_dt > mbrgm.creat_dttm
) then mbr_hist.meck
else mbrgm.facmbid
end as mb_fid,
.....
from
tempview1 mbrgm
left join left outer join tempview2 mbr_hist on (mbrgm.mrid = mbr_hist.mrid
and mbr_hist.efdt <= mbrgm.credttm
and mbr_hist.exdt > mbrgm.credttm
Every time I execute I get else condition value for mb_fid field i.e, mbrgm.facmbid. I have checked My data and compared with logic. As per logic it should go for then condition. I think while comparing mbr_hist.efdt <= mbr_pgm.credttm it is always not true.
I am having mbr_hist.efdt as a String type ex: 2017-07-22 21:58:46 and mbr_pgm.credttm as a timestamp ex:2011-08-13T11:00:00.910+0000. Is it like because of different in length of values ,my comparison is failing. What I can use to compare correctly.
Databricks can't directly compare the string with timestamp. You need to convert your string into the timestamp. By default, cast works only with strings in the ISO 8601 format, so you need to use the to_timestamp function with explicit date/time pattern to do the conversion.
like
select to_timestamp(mbr_hist.efdt, 'pattern') as efdt ...
I have a query that I am pulling in a department field, however, after a certain date I want this field to be populated as null.
For example, here is the code
Select T6.Segment2 as 'Old Department Code'
I do want this field to pull in the appropriate values, however after a certain date ( 04/01/2019 ) I want this field to show a NULL value.
Is this possible?
Not sure which DBMS you are using but it is basically the same for all of them when it comes to this... You want to use a CASE statement.
What this essentially does is it acts as an IF ELSE in your SELECT.
So in your case (ha, pun) (T-SQL Syntax):
SELECT
CASE
WHEN (YourDateFieldHere) < '04/01/2019' THEN (YourOutputFieldHere)
ELSE NULL
END (AS Alias)
FROM ...
CASE statements can check for multiple criteria, it doesn't have to just be one or the other, in this case just include more lines of WHEN (something) THEN (display this)
You can use case..when
( considering YYYYMMDD is the default format used in SAP at the internal level )
Select case when myDate >'20190104' then
null
else
T6.Segment2
end
as 'Old Department Code'
From yourTable
I have a column with datatype of varchar. I would like to replace all the values that are not numeric with NULL.
So for example my column can contain a value of MIGB_MGW but also 1352. The current expression I am using with Derived Column Transformation Editor is:
(DT_I4)kbup == (DT_I4)kbup ? 1 : 0
But of course this replaces all the values I want to keep with 1. What expression would I use to keep the numeric values? (1352 in this example)
If you want a null of varchar type, you can use NULL(DT_STR). For a DT_I4 you can use NULL(DT_I4) etc.
You can then use (DT_I4)kbup in place of your 1 to return the original varchar value that you want to keep, converted to a DT_I4:
(DT_I4)kbup == (DT_I4)kbup ? (DT_I4)kbup : NULL(DT_I4)
You could just convert them with a Derived Column and then use the ignore failure option in the Error output.
Use NOT LIKE
SELECT CASE
WHEN col NOT LIKE '%[^0-9]%' THEN col
ELSE NULL
END as Only_Numeric
FROM (VALUES ('MIGB_MGW'),
('1352')) tc(col)
Result :
Only_Numeric
------------
NULL
1352
Another option if 2012+ is Try_Convert()
SELECT Try_Convert(float,col)
FROM (VALUES ('MIGB_MGW'),
('2.6e7'),
('2.6BMW'),
('1352')) tc(col)
Returns
NULL
26000000
NULL
1352
I am loading a particular field that has date values. However, some of them are not complete... for example the values look like this
START_DATE
'2015-06-12'
'2016-12-24'
'2015-02' <--- this is what causes an error
'2016-01-03'
I have tried solving this by combining NULLIF with a LENGTH() function like so, but this is not allowed:
Start_date NULLIF LENGTH(:start_date)<10 to_date .....
this returns the error
Expecting positive integer or column name, found keyword length.
My main objective is to load dates that are of a proper format, and load NULL otherwise. What is the easiest way to do this within the ctl file? Can I avoid creating a custom function?
Say I have a table like this:
create table dateTable(START_DATE date)
and I need to load this file, where I want to insert NULL where the string does not match my pattern
'2016-12-28'
'2016-12-'
'2016-12-31'
I can add some logic in my ctl file to check the length of the string to load this way:
load data
infile dateTable.csv
into TABLE dateTable
fields enclosed by "'"
( START_DATE "to_date(case when length(:START_DATE) = 10 then :START_DATE end, 'yyyy-mm-dd')"
)
This simply checks the length of the string, but you can edit it anyway you need to build your own logic; notice that CASE gives NULL when no condition is matched, so this is equivalent to case when length(:START_DATE) = 10 then :START_DATE else NULL end.
This gives the following result:
SQL> select * from dateTable;
START_DATE
----------
28-DEC-16
31-DEC-16
In oracle, you can verify a string to make sure that is it valid date or not. Please Check IsDate function.
I want to set the value of a column based on another column's value by using Computed Column Specification in SQL Server.
Basically I want to set the columns' value to be a link if it has a result associated (in another column). This is what I tried putting in the formula part of that column but I always get this
Error Validating formula
T-SQL code:
SELECT
CASE
WHEN D.TestResultId IS NOT NULL
THEN ('Analysis')
ELSE 'Pending Analysis'
END
FROM DailyReport AS D
The query by itself is working fine, but when put in that into the computed column specification it always returns that error. I even tried creating a stored procedure and calling it, but still the same issue. Can I get any help on this?
You don't need the SELECT - FROM part on the computed column. It should just be:
CASE WHEN TestResultId IS NOT NULL
THEN ('<a href="Analysis?Result=' + CONVERT(varchar(max), TestResultId) +
'">Analysis</a>') ELSE 'Pending Analysis' END