I got SAS dataset into txt file format from client. But client didnt change the SAS date format to mm/dd/yyyy or yyyy-MM-dd. As SAS uses seconds since Jan 1, 1960, Date is coming like:
| response_dt |
+----------------
| 19724 |
| 19673 |
| 19698 |
| 19738 |
| 19738 |
I want to convert this to yyyy-MM-dd format in hive. Kindly help
Just read in the numbers. They are indeed days since Jan 1, 1960.
Then assign a format to them in one of the several ways you can, like
Data myData;
set myData;
format response_dt ddmmyy8.;
run;
If the dataset is huge, consider using proc datasets
Sas dates are days since 1jan1960.
i made this a wiki.
Can anyone add the correct function to add days to a date in hive?
Related
I have this code in Teradata that reads "x_date/100+190000". So from my understanding it removes the 'day' portion from DATE and then adds an INT number of days. Now I have to translate the same into BigQuery but can't see how.
edit: so what I have is a SELECT statement that includes the "x_date" field, which has a DATE format. It contains a list of dates in the form of 'yyyy-mm-dd'. The query reads something like:
SELECT x_date/100+190000
FROM x_table
and the field has this sort of rows:
| '2022-06-06' |
| '2020-03-06' |
| '2019-09-01' |
| '2028-05-06' |
What I don't understand exactly is what this functions are doing in Teradata.
My expected output should be in DATE format and should be copying (in BigQuery), whatever the Teradata function is doing to the field.
Use below
SELECT FORMAT_DATE('%Y%m', x_date)
FROM x_table
In my Oracle DB, I have a date field called HIGH_DATE. The format for some entries is "27-SEP-12" (DD-MON-YY) and for some entries it is "27-09-12" (DD-MM-YY).
Can someone help me in framing a select query through which I can get dates in either formats??
If you have a DATE column then it does not have any format; it is stored internally as 7-bytes (century, year-of-century, month, day, hour, minute, second) and it is only when the user interface being used to access the database returns data to the user that it then gets formatted (and all the dates will be implicitly converted to strings with a consistent format).
I'm going to assume that when you say:
I have a date field called "HIGH_DATE"
What you actually mean is: "I have a column with a VARCHAR2 data-type where I store date values".
If that is the case then all you need to do is:
SELECT TO_DATE( high_date, 'DD-MM-RR' ) AS high_date
FROM table_name;
Oracle's string-to-date conversion rules will match additionally the MON format model if you use the MM format model and don't specify an exact match using the FX format model.
If you have the test data:
CREATE TABLE table_name ( high_date ) AS
SELECT '23-09-20' FROM DUAL UNION ALL
SELECT '15-AUG-99' FROM DUAL;
Then the above query will output (depending on your NLS_DATE_FORMAT):
| HIGH_DATE |
| :------------------ |
| 2020-09-23T00:00:00 |
| 1999-08-15T00:00:00 |
db<>fiddle here
However, the best solution is going to be to stop storing the values as strings and to store them (without a format) as a date.
How to replace different dates such as Jan 11, 2003 | 6/17/1969 | 08/22/54
with a date string in “MM/DD/YY” format during creating table in hive
Hive won't understand 'Jan 11, 2003' it as a date string. Rest of them you can convert it into by using 'unix_timestamp()'.
I have columns with data type DATE in sparkSQL
e.g.
CREATE TABLE ABC(startDate DATE, EndDate DATE....
and I load data as LOAD DATA INPATH './input/user.txt' INTO TABLE ABC
In user.txt data is like
2016/06/12 2016/06/15
2016/06/12 2016/06/15
but it loads data as
null null
null null
if it's
2016-06-12 2016-06-15
2016-06-12 2016-06-15
then it takes the data correctly.
How to handle data when the date separator is '/ '?
I don't want to replace the separator in input file.
Please help me. Thanks.
I faced this issue before in Hive. I found a workaround for this. First load them as string instead of Data type DATE
ex:
CREATE TABLE ABC(startDate string, EndDate string....)
ROW FORMAT DELIMITED FIELDS TERMINATED BY ']'
STORED AS TEXTFILE
LOCATION './input/user.txt';
Then i used string functions to extract date/month/year from above fields. For example
select substr(date,1,4) as year,substr(date,6,1) as month .... from ABC
one other way is to replace the '/' with '-' and then cast them as DATE type and use Date functions
example
select regexp_replace(startDate,'/','-') from ABC
All the above is how to achieve it in Hive. To work on this in spark is also to first load them as string in to a dataframe.
val s1 = Seq(("2016/06/12", "2016/06/15" ), ("2016/06/12", "2016/06/15")).toDF("x", "y")
val result = s1.select(regexp_replace($"x","/", "-"),regexp_replace($"y","/", "-")).show()
result
+----------+----------+
| startDate| EndDate|
+----------+----------+
|2016-06-12|2016-06-15|
|2016-06-12|2016-06-15|
+----------+----------+
Hope this helps.
i know it's kinda late to answer this question but, in SPARK you can also include dateFormat in options while creating a table.
This will convert your date format from 2016/06/12 to 2016-06-12
CREATE TABLE IF NOT EXISTS ABC (
startDate DATE,
EndDate DATE,
...
)
using txt
options(
path "./input/user.txt",
dateFormat "yyyy/MM/dd"
)
select startDate, EndDate from ABC
result:
| startDate | EndDate |
|:----------|:---------|
|2016-06-12 |2016-06-15|
|2016-06-12 |2016-06-15|
I found one more way to do it using functions in SparkSQL on Spark 2.0 Preview Version
TO_DATE(from_unixtime(unix_timestamp(regexp_replace(startDate , '/','-'),'MM-dd-yyyy'))) AS startDate
Trying the change the date column from YYYYMMDD to MMDDYYYY while maintaining varchar value. Currently my column is set as varchar(10). Is there a way to change the strings in mass numbers because I have thousands of rows that need the format converted.
For example:
| ID | Date |
------------------------
| 1 | 20140911 |
| 2 | 20140101 |
| 3 | 20140829 |
What I want my table to look like:
| ID | Date |
------------------------
| 1 | 09112014 |
| 2 | 01012014 |
| 3 | 08292014 |
Bonus question: Would it cause an issue while trying to convert this column if there is data such as 91212 for 09/12/2012 or something like 1381 which is supposed to be 08/01/2013?
Instead of storing the formatted date in separate column; just correct the format while fetching using STR_TO_DATE function (as you said your dates are stored as string/varchar) like below. Again, as other have suggested don't store date data as string rather use the datetime data type instead
SELECT STR_TO_DATE(`Date`, '%m/%d/%Y')
FROM yourtable
EDIT:
In that case, I would suggest don't update your original table. Rather store this formatted data in a view or in a separate table all together like below
create view formatted_date_view
as
SELECT ID,STR_TO_DATE(`Date`, '%m/%d/%Y') as 'Formatted_Date'
FROM yourtable
(OR)
create table formatted_date_table
as
SELECT ID,STR_TO_DATE(`Date`, '%m/%d/%Y') as 'Formatted_Date'
FROM yourtable
EDIT1:
In case of SQL Server use CONVERT function like CONVERT(datetime, Date,110). so, it would be (Here 110 is the style for mm-dd-yyyy format)
SELECT ID,convert(datetime,[Date],110) as 'Formatted_Date'
FROM yourtable
(OR)
CAST function like below (only drawback, you can't use any specific style to format the date)
SELECT ID, cast([Date] as datetime) as 'Formatted_Date'
FROM yourtable
MS SQL Server Solution:
Which SQL are you trying with?
MSSQL Server 2008 R2
You can use Convert function on your date field. You have to specify the date's format Style.
For mm/dd/yyyy format Style value is 101.
Using with style value, your update statement can be:
UPDATE table_name
SET date = CONVERT( VARCHAR, date, 101 )
Refer To:
How to format datetime & date in Sql Server
SQL Server 2008 Date Format
Demo # MS SQL Server 2008 Fiddle
MySQL Solution:
it needs to stay in varchar or int and the dates are yyyymmdd and I need to change thousands of rows of data to be in mmddyyyy format.
Change to date type using str_to_date and then change again to string using date_format.
UPDATE table_name
SET date = DATE_FORMAT( STR_TO_DATE( date, '%Y%m%d' ), '%m%d%Y' )
The value 20140911 when converted from yyyymmdd to mmddyyyy format, will retain the leading 0 as 09112014.
Bonus question: Would it cause an issue while trying to convert this column if there is data such as 91212 for 09/12/2012 or something like 1381 which is supposed to be 08/01/2013
You can use str_to_date( '91212', '%c%e%y' ) to convert the same to valid date object. But MySQL, though defines to support single digit month and date numbers, it won't parse such date correctly and returns a NULL on such formats.
mysql> select str_to_date( '91212', '%c%e%y' ) s1, str_to_date( '091212', '%c%e%y' ) s2;
+------+------------+
| s1 | s2 |
+------+------------+
| NULL | 2012-09-12 |
+------+------------+
1 row in set, 1 warning (0.00 sec)
mysql> show warnings;
+---------+------+------------------------------------------------------------+
| Level | Code | Message |
+---------+------+------------------------------------------------------------+
| Warning | 1411 | Incorrect datetime value: '91212' for function str_to_date |
+---------+------+------------------------------------------------------------+
1 row in set (0.00 sec)