TL;DR - Is it possible to change the date format Sqoop uses?
I am importing data from a SQL Server using Sqoop (version 1.4.6) and am specifying a datetime column as my --check-column.
Sqoop is querying the database using dates in the format yyyy-MM-ddd hh:mm:ss.SSS. However, since SQL server is configured to use British dmy date format, it, counterintuitively, interprets a date beginning with a year as having the day in the second position rather than the month (wtf?!).
E.g SELECT ... WHERE modified < '2017-01-31 00:00:00.000' is interpreted as selecting the data where modified is less than the 1st day of the 31st month 2017 which obviously throws an error.
Is it possible to change the date format Sqoop uses?
A quick scan of the Sqoop documentation did not reveal a way to change the date format. The Microsoft Support article here suggests that you could avoid the problem by issuing a
SET DATEFORMAT 'ymd'
statement immediately after opening the connection, but it appears that Sqoop only supports that sort of operation for Oracle (oraoop-site-template.xml) and not for other JDBC drivers.
There is an mssql-jdbc pull request that, if accepted, would let you add ;connectionDateformat=ymd to your Sqoop connection URL. In the meantime you may just have to change the default language of the SQL login for your Sqoop job from "British English" to (US) "English":
Related
At the moment I have a column that holds timestamps as a datetime
The data is being stored as 10/30/2011 10:50:34 AM
My goal is to convert every date in that specific table to be formatted like 30 Oct 2011 10:50:34
The problem:
When I attempt to run this SQL it successfully processes.
UPDATE DatesTable
SET DateTime = '30 Oct 2011 10:50:34'
WHERE DateTime = '10/30/2011 10:50:34 AM'
I noticed that was weird so if I changed the SQL to say
UPDATE DatesTable
SET DateTime = '31 Oct 2011 10:50:34'
WHERE DateTime = '10/30/2011 10:50:34 AM'
It updates updates the datetime field with the correct day/time information but keeps it in the same format.
I have looked into other possible date datatypes for that column but nothing seems to support that date.
I have also come across this SQL but am unsure if it can solve my problem.
SELECT CONVERT(datetime, datecolumnname, formattingparam) as tmp FROM tablename
Converting Link
SQL Server doesn't store a DateTime in any string format - it's stored as an 8 byte numerical value.
The various settings (language, date format) only influence how the DateTime is shown to you in SQL Server Management Studio - or how it is parsed when you attempt to convert a string to a DateTime.
There are many formats supported by SQL Server - see the MSDN Books Online on CAST and CONVERT. Most of those formats are dependent on what settings you have - therefore, these settings might work some times - and sometimes not.
The way to solve this is to use the ISO-8601 date format that is supported by SQL Server - this format works always - regardless of your SQL Server language and dateformat settings.
The ISO-8601 format is supported by SQL Server comes in two flavors:
YYYYMMDD for just dates (no time portion) - note here: no dashes!, that's very important! YYYY-MM-DD is NOT independent of the dateformat settings in your SQL Server and will NOT work in all situations!
or:
YYYY-MM-DDTHH:MM:SS for dates and times - note here: this format has dashes.
This is valid for SQL Server 2000 and newer.
If you use SQL Server 2008 and the DATE datatype (only DATE - not DATETIME!), then you can indeed also use the YYYY-MM-DD format and that will work, too, with any settings in your SQL Server.
Don't ask me why this whole topic is so tricky and somewhat confusing - that's just the way it is. But with the YYYYMMDD format, you should be fine for any version of SQL Server and for any language and dateformat setting in your SQL Server.
The data is stored independent of format. You only need to reformat dates for display purposes.
sql server doesn't store the data in a text format. It is stored as a number.
Check out this page for how to format dates in sql
http://msdn.microsoft.com/en-us/library/ms187928.aspx
The datetime datatype tells the database engine how to store the data internally, so that you don't need to worry about human representations of the data. This is why your different UPDATE scripts don't change the format. The format is only applied when converting from the database's internal storage (a number representing the amount of time passed since a particular starting-point) and a human-readable date in the culture you require. You just need to format the date when it gets displayed, in the format suitable for your users. For example, for many, Month/Day/Year makes no sense.
Today I faced a problem while porting an SQL server database to H2DB. The SQL queries I had written to insert date time, wasn't working on H2DB. i.e '26-Jun-2019 01:00:00' gave exception on H2DB. My question is what is standard format for inserting date time which is database independent?
The standard format as defined by the SQL standard is the ISO format, prefixed with the keyword DATE, e.g.
DATE '2019-06-26'
or for a timestamp:
TIMESTAMP '2019-06-26 17:42:00'
However not all database products support the SQL standard in that regard, so you will most probably not find one single format that will work across all database products.
I have got several tables in my database (with data) that is formatted in the American standard of mm/dd/yyyy. Is there a way to convert the date to a British format (i.e. dd/mm/yyyy) that doesn't involve dropping and recreating the tables?
Thanks!
I set my data type to >date when I was creating my table. I can store dates in the format mm/dd/yyyy, but not dd/mm/yyyy.
As I've mentioned in my comment, dates are not stored with their display format - in fact, you can say that dates have no display format - only string representation of dates have a display format.
Whenever dealing with string literals representing date and datetime values in SQL server, use ISO 8601 datetime format (yyyy-MM-ddTHH:mm:ss or yyyyMMddTHHmmss).
SQL Server guarantees to properly parse this string representation into date / datetime values, without ambiguity.
Please note the T seperator between the date and the time. There is a very similar standard format, where the T is replaced with a white-space, but the DateTime data type have a bug parsing this format - and it is culture-dependent (Note that DateTime2 does not have that bug) - and that's another reason why you should never use datetime again.
When you use a string literal like '25/03/2018' it's easy for a human to see that it stands for March 25th 2018, but SQL Server will raise an error trying to parse this string into a date if the current value of DATEFORMAT is not DMY.
However, SQL Server will always parse ISO 8601 string representation of dates correctly, regardless of any local settings or previous set dateformat or set language statements etc'. '2018-02-01T15:40:50' will always be parsed is February 1st 2018, 3:40:50 PM.
Unless specified, As Martin Smith wrote in his comment, the default dateformat depends on the defualt language settings of the current login - so a query that works for one login might raise an error for another login - and that's another good reason never to trust culture-specific string representation of datetime.
DECLARE #dt DATETIME = '01/20/2019';
SELECT FORMAT( #dt, 'd', 'en-gb' ) AS 'UK'
Are you referring to the date format displayed by SQL Server Management Studio or a similar application? The format is controlled by Windows Control Panel settings, not by SQL Server. There is no internal format for dates in SQL Server.
This is defined by default from the machine where is running MS SQL Server.
To see all available cultures please do:
select * from sys.syslanguages
Then, you can change SQL Server language using:
SET LANGUAGE BRITISH
... and the date format will always be like you want.
Note: this will change all the database (not just the date format), the other way is to change the date format using the FORMAT function in T-SQL.
I am using Teradata JDBC 16.20 in Datagrip.
Whenever I try to do anything with date, it returns 1 day less.
For instance: SELECT date'2017-08-01' returns 2017-07-31 in Datagrip and in Teradata SQL Assistant it returns correctly 01/08/2017.
Does anyone know why?
We have the same issue using DataGrip with a Vertica database. My hunch is that the date is being shifted twice. When I select current_timestamp (which on Vertica is timestamp with timezone) the date is correct (typically one day earlier than the current date), but when I select current_date, I get the previous day (my timezone is US/Pacific). What I think is happening is the JDBC driver is adjusting the date based on the two timezones, and then DataGrip is then adjusting it a second time.
We don't have issues using the same Vertica database with the same JDBC driver, but a different SQL Client (DbVisualizer).
The only workaround I can offer using DataGrip is to install a much older version. DataGrip 2016.3.4, Build #DB-163.13906.13, built on February 21, 2017 seems to handle date columns correctly.
Adding -Duser.timezone=UTC to VM options seems to solve the problem.
At the moment I have a column that holds timestamps as a datetime
The data is being stored as 10/30/2011 10:50:34 AM
My goal is to convert every date in that specific table to be formatted like 30 Oct 2011 10:50:34
The problem:
When I attempt to run this SQL it successfully processes.
UPDATE DatesTable
SET DateTime = '30 Oct 2011 10:50:34'
WHERE DateTime = '10/30/2011 10:50:34 AM'
I noticed that was weird so if I changed the SQL to say
UPDATE DatesTable
SET DateTime = '31 Oct 2011 10:50:34'
WHERE DateTime = '10/30/2011 10:50:34 AM'
It updates updates the datetime field with the correct day/time information but keeps it in the same format.
I have looked into other possible date datatypes for that column but nothing seems to support that date.
I have also come across this SQL but am unsure if it can solve my problem.
SELECT CONVERT(datetime, datecolumnname, formattingparam) as tmp FROM tablename
Converting Link
SQL Server doesn't store a DateTime in any string format - it's stored as an 8 byte numerical value.
The various settings (language, date format) only influence how the DateTime is shown to you in SQL Server Management Studio - or how it is parsed when you attempt to convert a string to a DateTime.
There are many formats supported by SQL Server - see the MSDN Books Online on CAST and CONVERT. Most of those formats are dependent on what settings you have - therefore, these settings might work some times - and sometimes not.
The way to solve this is to use the ISO-8601 date format that is supported by SQL Server - this format works always - regardless of your SQL Server language and dateformat settings.
The ISO-8601 format is supported by SQL Server comes in two flavors:
YYYYMMDD for just dates (no time portion) - note here: no dashes!, that's very important! YYYY-MM-DD is NOT independent of the dateformat settings in your SQL Server and will NOT work in all situations!
or:
YYYY-MM-DDTHH:MM:SS for dates and times - note here: this format has dashes.
This is valid for SQL Server 2000 and newer.
If you use SQL Server 2008 and the DATE datatype (only DATE - not DATETIME!), then you can indeed also use the YYYY-MM-DD format and that will work, too, with any settings in your SQL Server.
Don't ask me why this whole topic is so tricky and somewhat confusing - that's just the way it is. But with the YYYYMMDD format, you should be fine for any version of SQL Server and for any language and dateformat setting in your SQL Server.
The data is stored independent of format. You only need to reformat dates for display purposes.
sql server doesn't store the data in a text format. It is stored as a number.
Check out this page for how to format dates in sql
http://msdn.microsoft.com/en-us/library/ms187928.aspx
The datetime datatype tells the database engine how to store the data internally, so that you don't need to worry about human representations of the data. This is why your different UPDATE scripts don't change the format. The format is only applied when converting from the database's internal storage (a number representing the amount of time passed since a particular starting-point) and a human-readable date in the culture you require. You just need to format the date when it gets displayed, in the format suitable for your users. For example, for many, Month/Day/Year makes no sense.