We are refactoring a weird system where within its backend, values that should be stored as number and date columns, with propperly designed constraints, were stored all as varchar (using only UI validation). The thing is, while we do this the new system our ones must start to run so:
Is it possible to manipulate this varchar columns as integer for say lower or greater comparisons and, given that the dates are stored with the same format dd/MM/yyyy is it possible to treat that also varchar column as a date and use functions as: IN BETWEEN?
You could add a computed column (or a view with a column) like:
CONVERT(DATE, column_name, 103)
For performance concerns you could persist and/or index the computed column.
However if you are going to change the structure may as well just fix the data type in the first place.
Would something like this work:
SELECT Column1, Column2
FROM Table
WHERE CONVERT(datetime, DateColumn, 3) BETWEEN #StartDate AND #EndDate
Related
I'm working on a table with a column, 'Expiry Date', as a varchar with all data formatted as DD/MM/YYYY.
The creator of the table has used the wrong type for this expiry date column and now the client needs to filter and show all records before and after the current date as the time. This means the type needs to be changed to date or datetime type to be able to use the CURDATE() function.
However, the current format of the values does not satisfy and wont allow the type to change unless the format is changed to YYYY-MM-DD (or similar).
Is there any way to mass format the values in this column and this column alone as there are thousands of entries and formatting one by one would be extremely time consuming.
Let me assume that you are using MySQL.
Perhaps the simplest method is to add a generated column that is a date:
alter table t add column expiry_date_date as
(str_to_date(expiry_date, '%d/%m/%Y'));
You can also fix the data:
update t
set expiry_date = str_to_date(expiry_date, '%d/%m/%Y');
This will implicitly convert the result of str_to_date() to a date, which will be in the YYYY-MM-DD format.
More importantly, you can then do:
alter table t modify column expiry_date date;
Here is a db<>fiddle.
You can do similar operations in other databases, but the exact code is a bit different.
What you need is an update on that column, but before doing it I suggest you to check if the result is what you want.
select replace(expiry_date, '/', '-') new_expiry_date
from table_name
If this returns the results you want you can run the following update:
update table_name
set expiry_date = replace(expiry_date, '/', '-')
Of course you will need to replace expiry_date and table_name with the names of your column and table.
I have an issue where I created a column sent_Date of type nvarchar while it's storing date and time.
Now when I try to sort it by date, it's not doing so correctly.
I am using this query:
select *
from tbl_skip
where sent_date > '9/27/2020 7:29:11 PM'
order by SENT_DATE desc
Like the comments have said, the real solution here is fix your design. That means changing the column's data type an nvarchar to a date and time data type, I'm going to use a datetime2(0) here, as your data is accurate to a second, so seems the most appropriate.
Firstly we need to convert the value to as ISO value. I'm also, however, going to create a new column called Bad_Sent_Date, to store values that could not be converted. Experience has taught many of us that systems that incorrectly use string data types to store dates (or numerical data) rarely have good data integrity rules on the value (because if they did, it wouldn't be an nvarchar) to start, so have bad values like '29/02/2019' or mix styles, such as having both '09/29/2020' and '29/09/2020'.
Based on the single example we have, I will assume your data is supposed to be in the format MM/dd/yy hh:mm:ss AM/PM:
ALTER TABLE dbo.tbl_skip ADD Bad_Sent_Date nvarchar(30) NULL;
GO
UPDATE TABLE dbo.tbl_skip
SET Bad_Sent_Date = CASE WHEN TRY_CONVERT(datetime2(0),Sent_date,101) IS NULL THEN Sent_date END,
Sent_Date = CONVERT(nvarchar(30),TRY_CONVERT(datetime2(0),Sent_date,101),126);
GO
Now we have an ISO format, we can change the table's data type:
ALTER TABLE dbo.tbl_skip ALTER COLUMN Sent_date datetime2(0) NULL;
Note that if you do have constraints on the column Sent_date, or it isn't NULLable, you will first need to DROP said CONSTRAINTs, change the column to be NULLable and then recreate said CONSTRAINTs after you have altered the column.
You can also review the "dates" that failed to be converted with the following:
SELECT bad_sent_date
FROM dbo.tbl_skip
WHERE bad_sent_date IS NOT NULL
AND Sent_date IS NULL;
Once that's all done, then your query simply needs an update to use an unambiguous date literal, and it'll work:
SELECT *
FROM tbl_skip
WHERE sent_date > '2020-09-27T19:29:11' --'9/27/2020 7:29:11 PM'
ORDER BY SENT_DATE DESC;
You can convert the data from string to datetime.
Please note i used 100 as an example to convert to date time. You can use below link to see if its behaving correctly. link -https://www.w3schools.com/sql/func_sqlserver_convert.asp
select *
from tbl_skip
where sent_date > convert(datetime,'9/27/2020 7:29:11 PM',100)
ORDER BY CONVERT(datetime,SENT_DATE,100) desc
You should be able to convert it to a datetime
select *
from tbl_skip
where sent_date > '9/27/2020 7:29:11 PM'
order by convert(datetime,SENT_DATE) desc
Just make sure the data in column is legit. If so, it would make sense to convert the column type to a datetime.
alter table tbl_skip alter column SENT_DATE datetime
If the data is mixed, you may need to fix it or use something like
order by try_convert(datetime,SENT_DATE) desc
How do i combine 2 column using SQL server 2005 ?
Problem is that The DateTime is stored in 1 column and the milliseconds is stored in another column.
I want to add the Milliseconds onto DateTime column to give it a more accurate DateTime.
I need to use this DateTime to query record accurate to milliseconds.
Any idea?
I need to replace DateTime with the added values.
SELECT *
FROM TABLENAME
WHERE [DateTime] >= '2011-04-12 12:00:00 AM'
AND [DateTime] <= '2011-05-25 3:35:04 AM'
and run the query.
Well, first solve your problem:
SELECT DATEADD(millisecond,<milliscolumn>,<datetimecolumn>) from <table>
And then file a bug report that these should just be stored in one column anyway.
You can do this, based on your sample query, but note that this destroys the possibility of the server being able to use an index:
SELECT * FROM TABLENAME WHERE
DATEADD(millisecond,[MillisecondColumn],[DateTime]) between
'2011-04-12T12:00:00' AND '2011-05-25T03:35:04'
If this is a large table, then indexes may be important. If you can't alter whatever's populating this data, you might want to add this calculation as a persisted computed column to this table, and then index and query against that.
Note that I've replaced your two comparisons with a single BETWEEN, and also adjusted the datetime strings so that they're not affected by regional settings.
SELECT (ColumnA + ColumnB) AS ColumnZ
FROM Table
might solve your problem.
Is it possible to use a function like DATEADD() or DATEPART() on a varchar column that may or may not hold something that looks like a datetime?
Currently I'm trying this by using a WHERE clause to select only rows that have datetime values in this column and getting the usual arithmetic overflow error.
I could perform some sort of operation to extract just the datetime values and put them in a temp table but I'm curious about this.
Example
SELECT DATEPART(wk, DateCreated) FROM tblOrders WHERE colType = 'datetime'
Don't worry about the table design, it's not as bad as that.
Don't worry about the table design, it's not as bad as that.
Seems unlikely, but...
Use IsDate
The question
Is it possible to ask SSIS to cast a value and return NULL in case the cast is not allowed instead of throwing an error ?
My environment
I'm using Visual Studio 2005 and Sql Server 2005 on Windows Server 2003.
The general context
Just in case you're curious, here is my use case. I have to store data coming from somewhere in a generic table (key/value structure with history) witch contains some sort of value that can be strings, numbers or dates. The structure is something like this :
table Values {
Id int,
Date datetime, -- for history
Key nvarchar(50) not null,
Value nvarchar(50),
DateValue datetime,
NumberValue numeric(19,9)
}
I want to put the raw value in the Value column and try to put the same value
in the DateValue column when i'm able to cast it to Datetime
in the NumberValue column when i'm able to cast it to a number
Those two typed columns would make all sort of aggregation and manipulation much easier and faster later.
That's it, now you know why i'm asking this strange question.
============
Thanks in advance for your help.
You could also try a Derived Column component and test the value of the potential date/number field or simply cast it and redirect any errors as being the NULL values for these two fields.
(1) If you just simply cast the field every time with a statement like this in the Derived Column component: (DT_DATE)[MYPOTENTIALDATE] - you can redirect the rows that fail this cast and manipulate the data from there.
OR
(2) You can do something like this in the Derived Column component: ISNULL([MYPOTENTIALDATE]) ? '2099-01-01' : (DT_DATE)[MYPOTENTIALDATE]. I generally send through '2099-01-01' when a date is NULL rather than messing with NULL (works better with Cubes, etc).
Of course (2) won't work if the [MYPOTENTIALDATE] field comes through as other things other than a DATETIME or NULL, i.e., sometimes it is a word like "hello".
Those are the options I would explore, good luck!
In dealing with this same sort of thing I found the error handling in SSIS was not specific enough. My approach has been to actually create an errors table, and query a source table where the data is stored as varchar, and log errors to the error table with something like the below. I have one of the below statements for each column, because it was important for me to know which column failed. Then after I log all errors, I do a INSERT where I select those records in SomeInfo that do not have an errors. In your case you could do more advanced things based on the ColumnName in the errors table to insert default values.
INSERT INTO SomeInfoErrors
([SomeInfoId]
,[ColumnName]
,[Message]
,FailedValue)
SELECT
SomeInfoId,
'PeriodStartDate',
'PeriodStartDate must be in the format MM/DD/YYYY',
PeriodStartDate
FROM
SomeInfo
WHERE
ISDATE(PeriodStartDate) = 0 AND [PeriodStartDate] IS NOT NULL;
Tru using a conditional split and have the records where the data is a date go along one path and the other go along a different path where they are updated to nullbefore being inserted.