I was testing Sprak 3 upgrades in AWS Athena and need to check date columns whether timestamp format is proper or not,Can any one please give me query to check whether date columns has any Values other than Timestamp format
Assuming that you have a varchar column you can try using date_parse wrapped in try:
select *
from table
where try(date_parse(string_column, 'your_expected_format')) is null -- assuming no original nulls in column
Or via try_cast for "standard" format:
select *
from table
where try_cast(string_column as timestamp) -- assuming no original nulls in column
Related
I'm working on a table with a column, 'Expiry Date', as a varchar with all data formatted as DD/MM/YYYY.
The creator of the table has used the wrong type for this expiry date column and now the client needs to filter and show all records before and after the current date as the time. This means the type needs to be changed to date or datetime type to be able to use the CURDATE() function.
However, the current format of the values does not satisfy and wont allow the type to change unless the format is changed to YYYY-MM-DD (or similar).
Is there any way to mass format the values in this column and this column alone as there are thousands of entries and formatting one by one would be extremely time consuming.
Let me assume that you are using MySQL.
Perhaps the simplest method is to add a generated column that is a date:
alter table t add column expiry_date_date as
(str_to_date(expiry_date, '%d/%m/%Y'));
You can also fix the data:
update t
set expiry_date = str_to_date(expiry_date, '%d/%m/%Y');
This will implicitly convert the result of str_to_date() to a date, which will be in the YYYY-MM-DD format.
More importantly, you can then do:
alter table t modify column expiry_date date;
Here is a db<>fiddle.
You can do similar operations in other databases, but the exact code is a bit different.
What you need is an update on that column, but before doing it I suggest you to check if the result is what you want.
select replace(expiry_date, '/', '-') new_expiry_date
from table_name
If this returns the results you want you can run the following update:
update table_name
set expiry_date = replace(expiry_date, '/', '-')
Of course you will need to replace expiry_date and table_name with the names of your column and table.
I am trying to read data from GP and ingest to HDFS using Spark. I need an integer column to partition the data which I read from the GP table.
The problem here is I don't have a primary column or any column that has unique values. In this scenario, column that I can rely on the most is the timestamp column where I can convert it to Integer/Long .
The data in timestamp column is present in the format:
select max(last_updated_timestamp) from schema.tablename => 2018-12-13 13:29:55
Could anyone let me know how can I cast the timestamp column including its milliseconds and produce an EPOCH value from it which I can use it in my spark code ?
You can use extract(epoch from last_updated_timestamp).
I have a column in Oracle DB which is varchar2 data type. Typical value stored in this column is like 06/16/2015 02:14:18 AM.
I am trying to get all records wherein this column is having records after 1st August 2015.
select *
from MYTABLE
where to_date(substr(MYCOLUMN,1,10),'dd-mm-yyyy') > to_date('01-08-2015','dd-mm-yyyy');
But, I am getting ORA-01843. Where am I doing wrong?
Respect the format in your VARCHAR
....where to_date(substr(MYCOLUMN,1,10),'mm/dd/yyyy')
I have a column in Oracle DB which is varchar2 data type. Typical value stored in this column is like 06/16/2015 02:14:18 AM.
The first question is why do you store DATE as string? Using appropriate data type is one of the most important part of database design and performance.
Understand that DATE doesn't have the format you see, it is internally stored in 7 bytes which is Oracle's proprietary format. Storing date as a string to have a fixed format is not recommended.
I would suggest first fix the design so that you don't have to do this overhead activity while comparing dates. In the longer run it will help you.
1. Add a new column as DATE data type.
ALTER TABLE table_name
ADD new_column DATE;
2. Update the new column.
UPDATE table_name
SET new_column = TO_DATE(old_column, 'mm/dd/yyyy hh:mi:ss pm');
3. DROP the old column.
ALTER TABLE table_name
DROP COLUMN old_column;
4. Rename the new column to old column name.
ALTER TABLE table_name
RENAME COLUMN old_name to new_name;
Now, you could compare dates easily:
SELECT * FROM MYTABLE WHERE mycolumn > to_date('01-08-2015','dd-mm-yyyy');
This will also use any regular index on the date column.
From performance point of view:
If you don't fix it now, you will keep facing performance issues. Because the immediate fix of SUBSTR will not let you use any regular index, you need to create a function-based index.
If in your table all values like 06/16/2015 02:14:18 AM then you can use trunc(to_date(MYCOLUMN,'mm/dd/yyyy HH:mi:SS PM'),'dd') against to_date(substr(MYCOLUMN,1,10),'dd-mm-yyyy').
My table table1 has the column date_txt which includes 2/16/2011 12:00:00 AM - column date_txt is VARCHAR2 (250 Char).
My table table1 also has the column date which is a DATE.
I would like to "update" my field:
The final output should be:
table1:
date
2/16/2011
So it takes from table1 date_txt the "date" and updates it to the column date as a date.
Any ideas? I am a bloody beginner.
You can use Oracle's to_date function to convert a string to a date:
update table1 set "date" = to_date(date_txt, 'MM/DD/YYYY HH:MI:ss AM')
See it working at SQL Fiddle.
column date_txt which includes 2/16/2011 12:00:00 AM
Firstly, DATE doesn't have any format. What you see is for display purpose to interpret date value easily.
Secondly, you should never ever store DATE as VARCHAR2. This is a big problem and a poor design.
Now that you have a bad design, it is a good idea to fix it right now.
Follow these steps:
Add a new column with DATE data type.
Update the new column with date values from the old column using TO_DATE.
Drop the old column.
Rename the new column to the old column.
I have already answered how to do it here https://stackoverflow.com/a/29625772/3989608
Try following:
update table1 set date=
(select to_date(date_txt,'dd/mm/yyyy') from table1 where id=yourId) where id=yourId;
Use primary key columns for ids.
If this is what you need everytime. I would suggest you to use Triggers
I just need to make a simple query of a table on a MapR cluster in that I want to know what the date is of the most recent record in the table. Dates are in a 'report_date' column in string format. I tried the following query without success:
select max(report_date) from partition.table_name
I know the second part of the statement works. Is there something wrong with the first part?
Thanks,
A
Your date column datatype is string hence the max function doesnt produce the output as desired.
for example : string column with values 1,2,3,4 and when you run max(column) you wont get the output as 4 , since max doesnt work on string datatype.
Try changing your datatype to DATE or TIMESTAMP , Which should work.
OR
if changing datatype is not possible then try,
If there is an auto incrementing ID column in the table or any column like so , then
select report_date from table_name order by ID desc.
This should provide you the max date sting.