Hi all i am new to hadoop i need to create hive UDF to convert string datatype Date to date datatype i have one table where date datatype is string
abc 2/9/2017
xyz 2/8/2017
city 2/7/2017
state 2/1/2017
UDF will convert date into yyy-mm-dd and insert into another table where date column datatype is Date
output like
abc 2017/2/9
xyz 2017/2/8
city 2017/2/7
state 2017/2/1
i have tried with
TO_DATE(from_unixtime(UNIX_TIMESTAMP(date,'yyyy-mm-dd')))
but got NULL value
dd/MM/yyyy
hive> select TO_DATE(from_unixtime(UNIX_TIMESTAMP('12/9/2017','dd/MM/yyyy')));
OK
_c0
2017-09-12
Related
I have a query that results in a timestamp value along with certain other calculations.
The result looks something like below -
City DateTime Value
London 2009-01-01 00:00:00.000000 22
New York 2010-01-01 00:00:00.000000 33
... ... ...
Is there any way to obtain the dateTime column with month and year - something like Jan-2009 and Jan-2010 instead of entire timestamp. I don't want to use the case statement.
t=# select now(),to_char(now(),'Mon-YYYY');
now | to_char
-------------------------------+----------
2017-09-20 07:49:34.360483+00 | Sep-2017
(1 row)
https://www.postgresql.org/docs/current/static/functions-formatting.html
to_char(timestamp, text)
and
https://www.postgresql.org/docs/current/static/functions-formatting.html#FUNCTIONS-FORMATTING-DATETIME-TABLE
for formats
the postgresql data type formating function to_char can solve this problem. It takes 2 arguments, a timestamp and a template pattern string, and return a date string according to the provided pattern. see Postgresql documentation for the complete pattern list.
You can try something like the following:
select city, to_char(your_date_field, 'Mon-YYYY'), value from your_table
I have columns with data type DATE in sparkSQL
e.g.
CREATE TABLE ABC(startDate DATE, EndDate DATE....
and I load data as LOAD DATA INPATH './input/user.txt' INTO TABLE ABC
In user.txt data is like
2016/06/12 2016/06/15
2016/06/12 2016/06/15
but it loads data as
null null
null null
if it's
2016-06-12 2016-06-15
2016-06-12 2016-06-15
then it takes the data correctly.
How to handle data when the date separator is '/ '?
I don't want to replace the separator in input file.
Please help me. Thanks.
I faced this issue before in Hive. I found a workaround for this. First load them as string instead of Data type DATE
ex:
CREATE TABLE ABC(startDate string, EndDate string....)
ROW FORMAT DELIMITED FIELDS TERMINATED BY ']'
STORED AS TEXTFILE
LOCATION './input/user.txt';
Then i used string functions to extract date/month/year from above fields. For example
select substr(date,1,4) as year,substr(date,6,1) as month .... from ABC
one other way is to replace the '/' with '-' and then cast them as DATE type and use Date functions
example
select regexp_replace(startDate,'/','-') from ABC
All the above is how to achieve it in Hive. To work on this in spark is also to first load them as string in to a dataframe.
val s1 = Seq(("2016/06/12", "2016/06/15" ), ("2016/06/12", "2016/06/15")).toDF("x", "y")
val result = s1.select(regexp_replace($"x","/", "-"),regexp_replace($"y","/", "-")).show()
result
+----------+----------+
| startDate| EndDate|
+----------+----------+
|2016-06-12|2016-06-15|
|2016-06-12|2016-06-15|
+----------+----------+
Hope this helps.
i know it's kinda late to answer this question but, in SPARK you can also include dateFormat in options while creating a table.
This will convert your date format from 2016/06/12 to 2016-06-12
CREATE TABLE IF NOT EXISTS ABC (
startDate DATE,
EndDate DATE,
...
)
using txt
options(
path "./input/user.txt",
dateFormat "yyyy/MM/dd"
)
select startDate, EndDate from ABC
result:
| startDate | EndDate |
|:----------|:---------|
|2016-06-12 |2016-06-15|
|2016-06-12 |2016-06-15|
I found one more way to do it using functions in SparkSQL on Spark 2.0 Preview Version
TO_DATE(from_unixtime(unix_timestamp(regexp_replace(startDate , '/','-'),'MM-dd-yyyy'))) AS startDate
Trying the change the date column from YYYYMMDD to MMDDYYYY while maintaining varchar value. Currently my column is set as varchar(10). Is there a way to change the strings in mass numbers because I have thousands of rows that need the format converted.
For example:
| ID | Date |
------------------------
| 1 | 20140911 |
| 2 | 20140101 |
| 3 | 20140829 |
What I want my table to look like:
| ID | Date |
------------------------
| 1 | 09112014 |
| 2 | 01012014 |
| 3 | 08292014 |
Bonus question: Would it cause an issue while trying to convert this column if there is data such as 91212 for 09/12/2012 or something like 1381 which is supposed to be 08/01/2013?
Instead of storing the formatted date in separate column; just correct the format while fetching using STR_TO_DATE function (as you said your dates are stored as string/varchar) like below. Again, as other have suggested don't store date data as string rather use the datetime data type instead
SELECT STR_TO_DATE(`Date`, '%m/%d/%Y')
FROM yourtable
EDIT:
In that case, I would suggest don't update your original table. Rather store this formatted data in a view or in a separate table all together like below
create view formatted_date_view
as
SELECT ID,STR_TO_DATE(`Date`, '%m/%d/%Y') as 'Formatted_Date'
FROM yourtable
(OR)
create table formatted_date_table
as
SELECT ID,STR_TO_DATE(`Date`, '%m/%d/%Y') as 'Formatted_Date'
FROM yourtable
EDIT1:
In case of SQL Server use CONVERT function like CONVERT(datetime, Date,110). so, it would be (Here 110 is the style for mm-dd-yyyy format)
SELECT ID,convert(datetime,[Date],110) as 'Formatted_Date'
FROM yourtable
(OR)
CAST function like below (only drawback, you can't use any specific style to format the date)
SELECT ID, cast([Date] as datetime) as 'Formatted_Date'
FROM yourtable
MS SQL Server Solution:
Which SQL are you trying with?
MSSQL Server 2008 R2
You can use Convert function on your date field. You have to specify the date's format Style.
For mm/dd/yyyy format Style value is 101.
Using with style value, your update statement can be:
UPDATE table_name
SET date = CONVERT( VARCHAR, date, 101 )
Refer To:
How to format datetime & date in Sql Server
SQL Server 2008 Date Format
Demo # MS SQL Server 2008 Fiddle
MySQL Solution:
it needs to stay in varchar or int and the dates are yyyymmdd and I need to change thousands of rows of data to be in mmddyyyy format.
Change to date type using str_to_date and then change again to string using date_format.
UPDATE table_name
SET date = DATE_FORMAT( STR_TO_DATE( date, '%Y%m%d' ), '%m%d%Y' )
The value 20140911 when converted from yyyymmdd to mmddyyyy format, will retain the leading 0 as 09112014.
Bonus question: Would it cause an issue while trying to convert this column if there is data such as 91212 for 09/12/2012 or something like 1381 which is supposed to be 08/01/2013
You can use str_to_date( '91212', '%c%e%y' ) to convert the same to valid date object. But MySQL, though defines to support single digit month and date numbers, it won't parse such date correctly and returns a NULL on such formats.
mysql> select str_to_date( '91212', '%c%e%y' ) s1, str_to_date( '091212', '%c%e%y' ) s2;
+------+------------+
| s1 | s2 |
+------+------------+
| NULL | 2012-09-12 |
+------+------------+
1 row in set, 1 warning (0.00 sec)
mysql> show warnings;
+---------+------+------------------------------------------------------------+
| Level | Code | Message |
+---------+------+------------------------------------------------------------+
| Warning | 1411 | Incorrect datetime value: '91212' for function str_to_date |
+---------+------+------------------------------------------------------------+
1 row in set (0.00 sec)
I have a table of data imported from CSV as follows:
FirstTimeTaken LatestTimeTaken Market Outcome Odds NumberOfBets VolumeMatched InPlay
03/08/2013 15:30:14 03/08/2013 15:32:28 Over/Under 3.5 Goals Over 3.5 Goals 5 10 118 1
03/08/2013 14:26:40 03/08/2013 14:29:43 Correct Score 0 - 0 7 12 279 1
03/08/2013 15:15:34 03/08/2013 15:27:39 Match Odds Barnsley 110 7 9 1
28/07/2013 16:57:26 29/07/2013 21:35:55 Match Odds Barnsley 3 9 35 0
I had to import the first 2 columns in varchar format because I was getting errors trying to import as datetime. Now I have the data in a table, I need to convert the Column format from Varchar to Datetime. I tried:
ALTER TABLE #CSVTest_Data
ALTER COLUMN FirstTimeTaken DATETIME
ALTER TABLE #CSVTest_Data
ALTER COLUMN LatestTimeTaken DATETIME
This results in error: 'The conversion of a varchar data type to a datetime data type resulted in an out-of-range value'.
I know that removing the last row of data gets rid of the problem, so I suspect that the system thinks the date format is MM/DD/YYYY whereas it is actually DD/MM/YYYY.
The following query works fine:
SELECT convert(VARCHAR(50),FirstTimeTaken,105) from #CSVTest_Data
SELECT convert(VARCHAR(50),LatestTimeTaken,105) from #CSVTest_Data
But this does not convert the column format to datetime. I would appreciate some help on how to do this. Thanks.
Try using SET DATEFORMAT.
SET DATEFORMAT dmy
You can select the data from your #Temp table as follows:
SELECT
CONVERT(DATETIME, FirstTimeTaken, 103 ),
CONVERT(DATETIME, LatestTimeTaken, 103)
FROM #CSVTest_Data
This returns the fields as DATETIME data types. From there, do whatever you need to.
How to get the formatted date in SQL Server CE?
I have a column on a table that contains Date, but the column type is nvarchar
ID Date
----------------------
1 05/08/2012
2 10/08/2012
3 05/10/2012
The date format is MM/dd/yyyy, but it is in nvarchar.
Also, I want to have a WHERE clause to select the ID on a specific date. How can I do that? This is for SQL Server CE. Thank you very much.
you need to cover it use convert or cast function ...
cast(datecolumn as DateTime)
or
CONVERT(datetime,datecolumn,101)
For current date, you can use:
SELECT CONVERT(nvarchar,GETDATE(),101)
Output: (As of answered date)
05/31/2016