IN Netezza , I am trying to check if date value is valid or not ; something like ISDATE function in SQL server.
I am getting dates like 11/31/2013 which is not valid, how in Netezza I can check if this date is valid so I exclude them from my process.
Thanks
I don't believe there is a built-in Netezza function to check if a date is valid. You may be able to write a LUA function to do this, or you could try joining to a "Date" lookup table, like so:
Create a table with two columns:
DATE_VALUE date
DATE_STRING varchar(10)
Load data into this table for valid dates (generate a file in your favorite tool, excel, unix, whatever). There can even be more than one row per DATE_VALUE (different "valid" formats) if all you use this for is this check. If you fill in from, say, 1900 to 2100, as long as your data is within that range, you'll be fine. And it's a small table, too, for ~200 years only ~7300 rows. Add more if needed. Heck, since the NZ date datatype goes from AD1 to AD 9999, you could fill it completely with only 3.4 million rows (small for NZ).
Then, to isolate rows that have invalid dates, just use a JOIN or an EXISTS / NOT EXISTS to this table, on DATE_STRING. Since the table is so small, netezza will likely broadcast it to all SPUs, making the performance impact trivial.
Netezza Analytics Package 3.0 (free download) comes with a couple LUA functions that verify date values: isdate() and todate(). Very simple to install / compile.
Related
I want to start by saying my SQL knowledge is limited (the sololearn SQL basics course is it), and I have fallen into a position where I am regularly asked to pull data from the SQL database for our ERP software. I have been pretty successful so far, but my current problem is stumping me.
I need to filter my results by having the date match from 2 separate tables.
My issue is that one of the tables outputs DATETIME with full time data. e.g. "2022-08-18 11:13:09.000"
While the other table zeros the time data. e.g. "2022-08-18 00:00:00.000"
Is there a way I can on the fly convert these to just a DATE e.g. "2022-08-18" so I can set them equal and get the results I need?
A simple CAST statement should work if I understand correctly.
CAST( dateToConvert AS DATE)
I want to create a column of data type having only 'mm-dd' values.
Is it possible and if yes how should I do it?
Note: Instead of "2022-06-07", I want "07-06"
There is no date type that can store that format - in fact none of the date types store a date and/or time in any of the formats you typically recognize.
For your specific requirement, that looks like a char(5) for the data type, but how you constrain it so that it will only accept valid date values, I have no idea. You'd think this would work:
CHECK (TRY_CONVERT(date, string_column + '-2022', 105) IS NOT NULL)
But what about leap years? February 29th is sometimes valid, but you've thrown away the only information that can make you sure. What a bunch of mess to store your favorite string and trust that people aren't putting garbage in there.
Honestly I would store the date as a date, then you can just have a computed column (or a column in a view, or just do this at query time:
d_slash_m_column AS CONVERT(char(5), date_column, 105)
Why not just in your query (or only in a view) say:
[output] = CONVERT(char(5), data_in_the_right_type, 105)
?
I'd personally stay away from FORMAT(), for reasons I've described here:
FORMAT() is nice and all, but…
FORMAT is a convenient but expensive function - Part 1
FORMAT is a convenient but expensive function - Part 2
You can use the SQL Server FORMAT function:
FORMAT(col1, 'dd/MM')
Check the demo here.
In such cases using char or varchar is not the best option as in those cases the underlying DB constraints that validate the integrity of the data do not kick in.
Best option is to use an arbitrary year and then put in a proper date, so for example for storing 01-Jan, the db column should store proper date with year as any arbitrary value, e.g. 2000. So your db should say 2000-01-01.
With such a solution you are still able to rely on the DB to raise an error if you tried month 13. Similarly sorting will work naturally as well.
I have written a program that logs events into a db file as they happen using SQL statement. Whenever I open the table to view, I specifically request the dataset in descending order based on ALARM's date and time. It seems to work only for part of the table. I am using SQLite3 and my program is written in delphi or Pascal.
Here is the SQL statement:
SELECT *
FROM Alarms
ORDER BY datetime(ALARMTIME) DESC
Here is a snapshot of the table. Pay attention to the red arrows. That's where Alarm's date and time doesn't follow descending order. I don't know why this is happening.
I'm not sure how you created your date/time string in your database since that information wasn't given in the question. However, according to the documentation for the datetime() function, the format you have of MM/DD/YYYY HH:MM:SS xx is not one of the accepted formats. In fact, if you do SELECT datetime('1/23/2018 01:40:00 PM') at an SQLite prompt, you get NULL. Whereas, if you use an acceptable format, you do not: SELECT datetime('2018-01-23') gives '2018-01-23 00:00:00'.
So I think the solution is to write the ALARMTIME field using datetime('now'), which does yield a format accepted by datetime(). If you need to sort based upon the format you have now, you can't do it with datetime(). You'd need to reformat the field using string functions to get it in a form you could do a string compare with (e.g., as is done here).
The OP has indicated in a comment that the ALARMTIME is set using the following Pascal code:
FieldByName('AlarmTime').AsDateTime := now;
There's no guarantee that Pascal is going to use a date/time string format in this context that is compatible with SQLite's datetime() function. So Pascal date/time formatting functions can be used to create a format more specifically acceptable by SQLite's datetime(). Then you'd use something like:
FieldByName('AlarmTime').AsString := FormatDateTime('YYYY-MM-DD hh:nn:ss',now);
Now this will change the default view of the date to YYYY-MM-DD .... If you still want your view of the table to show MM/DD/YYYY... then you'll either need to go back to my prior comment about processing the string on the fly in the comparison for sort, or write a little view formatting code so that it displays in a format that's different than what is stored internally, which is a common view/model separation technique.
If you can write your original ALARMTIME format as MM/DD/YYYY and make sure you pre-pad with zeroes (e.g., 01/09/2018 not 1/9/2018) then you can use SQLite's substr function:
SELECT * FROM Alarms
ORDER BY (substr(ALARMTIME,7,4)||substr(ALARMTIME,1,2)||substr(ALARMTIME,4,2)||substr(ALARMTIME,11)) DESC
And you would create your ALARMTIME using this:
FieldByName('AlarmTime').AsString := FormatDateTime('dd-mm-yyyy hh:nn:ss', now);
The above solutions are fairly generic. Depending upon the client library you are using (which you have not specified), there may be another more suitable approach to solving the problem.
You cannot do, for example, SELECT * FROM Alarms ORDER BY ALARMTIME DESC and get an accurate sort since, for example, the date 12/1/2018 would come after 2/1/2018 in that sort ordering even though 2/1/2018 is later in time. This is because 2 follows 1 in the ASCII collating sequence.
If you need to keep your current ALARMTIME string format and not change how you're saving it, which is somewhat free form m/d/yyyy in which the day or month can have one or two digits, you're going to have a bit of work to do in order to sort it if your client library doesn't support some helpers in this regard. Perhaps your only other option would be to use a custom SQLite function. These are written in C and compiled and linked with SQLite. You'd have to find one already written, or write your own.
I have a project that I am working on that requires me to delete records from the database if they are atleast 3 years old.
I have something like this in DB2 SQL to get the date:
SELECT * FROM tableA
WHERE ADD_DATE < CHAR(CURRENT DATE-3 YEARS)
ADD_DATE is stored as Characters in my system, this is why I am converting
I know it is also possible to get the date and format it in VB.net which is the language I am using to call the SQL statements.
My question is whether it would be faster/better to get the date and perform the conversion inside the SELECT in SQL or would it be better to get the current date and convert it in VB.net and then use that date in the SQL statement. I'm thinking VB.net would be better because there are thousands of records that must be compared. I should be able to set it up in VB so that it only retrieves the date and converts it once but I am not sure what kind of performance hit each takes from these statements.
Thanks in advance.
If all you are doing with a call to the database would be getting the date, then it would be faster to get it client-side and avoid the round-trip to the database.
If you do it server side and you're comparing your date in a single set-based operation then the time difference for that is negligible. If you do the check in something loop-based (a cursor or something) then you'll be wasting time.
It doesn't sound like this is applicable to you, but for future reference be sure to take into consideration the possibility of the client and the database server being in different timezones. It could be safer to do it one way or the other based on the time zone your data is generated for.
Doing a "Now" in VB.Net will definitely be faster than hitting the database.
I'd like to know what my best option would be to import data from an excel file on a weekly or monthly basis. At first, I thought I would use SSIS, but after much struggle with seemingly simple tasks, I'm starting to rethink my plan. Would it be better/easier to just write the SQL by hand or use the services of an SSIS package? The basic process will be as follows:
A separate process will download an .xls file to a local fileshare.
The xls file will have a filename like: 'myfilename MON YY'.
I will need to read the month and year from the the filename, reformat it to a sql date and then query a DimDate table to find the corresponding date key.
For each row (after the first 2 header rows), insert the data with the date key, unless the row is a total row, then ignore.
Here are some of the issues I've been encountering with SSIS:
I can parse the date string from a flat file datasource, but can't seem to do it with an excel data source. Also, once parsed, i cannot seem to convert the string to a date in order to perform the lookup for the date key. For example, I want to do something like this:
select DateKey from DimDate
where ActualDate = convert(datetime, '01-' + 'JAN-10', 120)
but i don't think it is possible to use the 'convert' or 'datetime' keywords in an expression builder. I have been also unable to find where I can edit the SQL to ignore the first 2 rows of data.
I'm very skeptical of using SSIS because it seems like a Kludgy way of doing something that can probably be accomplished more efficiently writing the SQL yourself, but I may be forced to use SSIS. Thoughts?
SSIS is definitely the direction to go.
To hit on your problems: (DT_DBTIMESTAMP) is the conversion you want. The syntax is a bit different. For instance to convert your example date I would use:
(DT_DBTIMESTAMP)"01/01/2010"
If you use that expression in a derived column to replace your string date (or create a new column), you could then do a lookup against datetime columns in a DB.
If you need to exclude the first two rows, you will either need to write an SQL statement to query the file (as opposed to an excel file reader source) or use a conditional split to throw them away based on any condition that can be repeated with every import.
Flat files easier to work with, and do allow you to throw away x number of initial rows.