I have below values coming from a flat file which may contain single digit month & day field:
9/14/2020 07:20:18.630000
7/7/2020 16:24:57.700000
10/24/2019 03:40:52.380000
11/9/2020 20:21:32.420000
Now I need to load this to a column having TIMESTAMP(6) as the data type.
Can someone please help on this? I am using TD SQL Assistant version 16.
SQL Assistant is not a load utility, e.g. TPT fully supports dealing with intput like this.
Your other post shows that you already use a RegEx to add the missing zeroes and you apply the correct format. This is indicating bad data in your input. You might try to spot the error in the input file (check how many rows have been loaded and check the following lines).
Or you apply TRYCAST which doesn't fail, but returns a NULL for bad dates. But yikes, it doesn't support FORMAT, thus you must rearrange the MDY to YMD first:
trycast(RegExp_Replace(RegExp_Replace(x,'\b([0-9])\b', '0\1'), '(..).(..).(....)(.*)','\3-\1-\2\4') as timestamp(6))
Related
I have a two tables in a database in AWS Athena that I want to join.
I want to join them by several columns, one of them being date.
However in one data set the date string is encoded for single value months is encoded as
"08/31/2018"
While the other would have it encoded as
"8/31/2018"
Is there a way to make them the same format?
I am unsure if it is easier to add the extra 0 to strings which have lack the extra 0 or to concatenate strings which have the extra 0.
Based on what I have researched I think I will have to use the CASE and CONCAT functions.
Both of the tables were loaded into the database from a CSV file, and the variables are in the string format.
I have tried changing the values manually in the CSV file, tried running an R script on one of the tables to format the date in the same way, and have also tried re-loading the tables into the database as the same date format.
However no matter what I do whenever it is loaded into the database, even when they have the same date type, it always loads them with different formats.
One with the the extra 0 and the other without it.
The last avenue I haven't tried is through a SQL query.
However I am not well versed in Athena and am having a hard time formatting this query.
I know this is rather vague, so please ask me for more information if you need.
If someone could help me start this query I would be grateful.
Thank you for the help.
Here is the query for changing dates in Athena.
date_parse(table.date_variable,'%m/%d/%Y')
Though Athena tables are immutable once created.
You can convert the value to date using date_parse(). So, this should work:
date_parse(t1.datecol, '%m/%d/%Y') = str_to_date(t2.datecol, '%m/%d/%Y')
Having said that, you should fix the data model. Store dates as dates not as strings! Then you can use an equality join and that is just better all around.
I have written a program that logs events into a db file as they happen using SQL statement. Whenever I open the table to view, I specifically request the dataset in descending order based on ALARM's date and time. It seems to work only for part of the table. I am using SQLite3 and my program is written in delphi or Pascal.
Here is the SQL statement:
SELECT *
FROM Alarms
ORDER BY datetime(ALARMTIME) DESC
Here is a snapshot of the table. Pay attention to the red arrows. That's where Alarm's date and time doesn't follow descending order. I don't know why this is happening.
I'm not sure how you created your date/time string in your database since that information wasn't given in the question. However, according to the documentation for the datetime() function, the format you have of MM/DD/YYYY HH:MM:SS xx is not one of the accepted formats. In fact, if you do SELECT datetime('1/23/2018 01:40:00 PM') at an SQLite prompt, you get NULL. Whereas, if you use an acceptable format, you do not: SELECT datetime('2018-01-23') gives '2018-01-23 00:00:00'.
So I think the solution is to write the ALARMTIME field using datetime('now'), which does yield a format accepted by datetime(). If you need to sort based upon the format you have now, you can't do it with datetime(). You'd need to reformat the field using string functions to get it in a form you could do a string compare with (e.g., as is done here).
The OP has indicated in a comment that the ALARMTIME is set using the following Pascal code:
FieldByName('AlarmTime').AsDateTime := now;
There's no guarantee that Pascal is going to use a date/time string format in this context that is compatible with SQLite's datetime() function. So Pascal date/time formatting functions can be used to create a format more specifically acceptable by SQLite's datetime(). Then you'd use something like:
FieldByName('AlarmTime').AsString := FormatDateTime('YYYY-MM-DD hh:nn:ss',now);
Now this will change the default view of the date to YYYY-MM-DD .... If you still want your view of the table to show MM/DD/YYYY... then you'll either need to go back to my prior comment about processing the string on the fly in the comparison for sort, or write a little view formatting code so that it displays in a format that's different than what is stored internally, which is a common view/model separation technique.
If you can write your original ALARMTIME format as MM/DD/YYYY and make sure you pre-pad with zeroes (e.g., 01/09/2018 not 1/9/2018) then you can use SQLite's substr function:
SELECT * FROM Alarms
ORDER BY (substr(ALARMTIME,7,4)||substr(ALARMTIME,1,2)||substr(ALARMTIME,4,2)||substr(ALARMTIME,11)) DESC
And you would create your ALARMTIME using this:
FieldByName('AlarmTime').AsString := FormatDateTime('dd-mm-yyyy hh:nn:ss', now);
The above solutions are fairly generic. Depending upon the client library you are using (which you have not specified), there may be another more suitable approach to solving the problem.
You cannot do, for example, SELECT * FROM Alarms ORDER BY ALARMTIME DESC and get an accurate sort since, for example, the date 12/1/2018 would come after 2/1/2018 in that sort ordering even though 2/1/2018 is later in time. This is because 2 follows 1 in the ASCII collating sequence.
If you need to keep your current ALARMTIME string format and not change how you're saving it, which is somewhat free form m/d/yyyy in which the day or month can have one or two digits, you're going to have a bit of work to do in order to sort it if your client library doesn't support some helpers in this regard. Perhaps your only other option would be to use a custom SQLite function. These are written in C and compiled and linked with SQLite. You'd have to find one already written, or write your own.
I am having to create a second header line and am using the first record of the Query to do this. I am using a UNION All to create this header record and the second part of the UNION to extract the Data required.
I have one issue on one column.
,'Active Energy kWh'
UNION ALL
,SUM(cast(invc.UNITS as Decimal (15,0)))
Each side are 11 lines before and after the Union and I have tried all sorts of combinations but it always results in an error message.
The above gives me "Error converting data type varchar to numeric."
Any help would be much appreciated.
The error message indicates that one of your values in the INVC table UNITS column is non-numeric. I would hazard a guess that it's either a string (VARCHAR or similar) column or something else - and one of the values has ended up in a state where it cannot be parsed.
Unfortunately there is no way other than checking small ranges of the table to gradually locate the 'bad' row (i.e. Try running the query for a few million rows at a time, then reducing the number until you home in on the bad data). SQL 2014 if you can get a database restored to it has the TRY_CONVERT function which will permit conversions to fail, enabling a more direct check - but you'll need to play with this on another system
(I'm assuming that an upgrade to 2014 for this feature is out of the question - your best bet is likely just looking for the bad row).
The problem is that you are trying to mix header information with data information in a single query.
Obviously, all your header columns will be strings. But not all your data columns will be strings, and SQL Server is unhappy when you mix data types this way.
What you are doing is equivalent to this:
select 'header1' as col1 -- string
union all
select 123.5 -- decimal
The above query produces the following error:
Error converting data type varchar to numeric.
...which makes sense, because you are trying to mix both a string (the header) with a decimal field.
So you have 2 options:
Remove the header columns from your query, and deal with header information outside your query.
Accept the fact that you'll need to convert the data type of every column to a string type. So when you have numeric data, you'll need to cast the column to varchar(n) explicitly.
In your case, it would mean adding the cast like this:
,'Active Energy kWh'
UNION ALL
,CAST(SUM(cast(invc.UNITS as Decimal (15,0))) AS VARCHAR(50)) -- Change 50 to appropriate value for your case
EDIT: Based on comment feedback, changed the cast to varchar to have an explicit length (varchar(n)) to avoid relying on the default length, which may or may not be long enough. OP knows the data, so OP needs to pick the right length.
I need verify that all cells in column contain data in only date format. How it possible to verify?
*I think it isn't LIKE function.
DATE doesn't have any format. What you see is for display purpose so that it could be easily interpreted.
DATE datatype is stored in a proprietary format internally in 7 bytes. It is a bad idea and makes no sense to verify the format while date is stored in an internal format. As I said, format is only for display.
If the date column is not a DATE data type, then it is a design flaw. And, any application based on such a flawed database design is on the verge to break anytime.
Storing DATE values other than date data type is just like not understanding the basics.
You should first fix the design to get a permanent solution. Any solution to your question is just another workaround.
Let me show a small example how it creates even more confusion.
The following date :
01/02/2015
Is it:
1st Feb 2015 or,
2nd Jan 2015
There is no way to tell that. It could be either DD or MM. This being just one among so many other problems due to the incorrect data type.
Store date values as DATE data type only, period.
Based on your last question, I think you are looking for something like this:
SELECT COUNT(*) FROM ...
WHERE NOT REGEXP_LIKE (A, '^XXX/MOSCOW/XXXMSX/[0-9]{4}-[0-9]{2}-[0-9]{2}$')
If count is greater than zero, something doesn't match. If you want more detail on what doesn't match, change your SELECT clause appropriately.
If you are looking for multiple date formats, you can change your regular expression appropriately. The | operator in most flavors of regular expression, including Oracle's, lets you define multiple patterns in the same space. You might use something like
SELECT COUNT(*) FROM ...
WHERE NOT
REGEXP_LIKE (A,
'^XXX/MOSCOW/XXXMSX/[0-9]{4}-[0-9]{2}-[0-9]{2}$|^[0-9]{4}-[0-9]{2}-[0-9]{2}$')
adding as many different matching patterns as you need.
Try
SELECT *
FROM POL
WHERE NOT REGEXP_LIKE(TR_KRY, '^(0[1-9]|([1-2][0-9])|30|31)-(([0][1-9])|10|11|12)-[0-9]{4}$')
This will return you all rows where TR_KRY is not formatted as 'DD-MM-YYYY', where DD is '01'-'31', MM is '01'-'12', and YYYY is any four numeric digits.
As others have said, storing dates as character strings is not a good idea. In the field you're looking at, it might be that the date is stored as DD-MM-YYYY (day-month-year - the usual case in Europe and perhaps elsewhere), or it might be that the date is stored as MM-DD-YYYY (month-day-year - a common practice in the US). If possible, I suggest you should convert this field to the DATE data type so that the TO_CHAR function can be used to produce a text version of the date in whatever format is desired.
Given the example data you've shown in comments (and that's also not good practice - you should go back and edit the question when you want to include additional information) it appears the dates are formatted as DD-MM-YYYY and I've set up the regular expression above to deal with this as best as possible.
There is a problem in VBA text box while filling input mask property:
I am trying to make the combination of date and time:
Hence i put it like below:
00/00/00;0;_00:00;0;_
But while running the application, i am only getting 00/00/00 (Date).
But i remember, i got the result as like 00/00/00 00.00 as expected when i first put the expression as like above;
but now i am not getting it :-(
The InputMask property can contain up to three sections separated by semicolons (;)
Your mask should be like this:
"00/00/00 00:00;0;0"
or
"00/00/00 00:00;0;_" // to display it like __/__/__ __:__
Why not just use the built in "General Date" format? I've found over the years that input masks are very restricting and basically a pain. Although it's been so long since I've used them that I don't recall the details of why I despise them.
This also has the benefit of respecting the users choices of regional date format. For example I always use yyyy-mm-dd format.
Also a client had a situation where the date format was decreed to be Medium Date on all fields. Which is dd-mmm-yy. It later turned out that in a table of 100K records there were twelve dates before 1900. They had simple had something extra keyed in in the year so Windows/Access interpreted those dates as being in the 3rd or 5th century or whatever. Now these dates weren't used in any kind of calculation so it wasn't a big deal. SQL Server upsizing to small date/time fields didn't appreciate those though.