In our current database, we have several columns of type datetime that are populated with a default constraint of GETDATE(). GATDATE() populates the column with the system time taking into consideration BST (We're in UK).
I need a script that will convert all the columns from BST to UTC. I am a java dev, and I could come up with a crude method, but we have a short downtime window with which to apply the change so as optimised an approach as I can take is needed.
In this case, a faster, more convoluted method would win hands down over a clean or elegant solution (not that the two need to be mutually exclusive!).
Any suggestions on methodology, or of course implementable sql, would be very gratefully received!
(Obviously the constraints will also be changed to instead use SYSUTCDATETIME()
I googled around and found no answer. The question is whether for an existing SQL table (assume any of H2, MySQL, or Postgres)...
Is there a way to get the last-update timestamp value for a given table row. That is, without explicitly declaring a new column (altering the table), and/or adding triggers that update a timestamp column.
I'm using a JDBC driver, preparing statements, getting ResultSets and so forth. I need to be able to determine whether the data has changed recently or not, and for this a timestamp would help. If possible I want to avoid adding timestamp columns across all tables in the system.
There is no implicit standard approach to this problem. The standard way is having an explicit column and logic in a db trigger or app function...
As mentioned, there are ways to do it through the logs but it's hard and usually wont be much accurate. For example, in Postgres you can enable commit timestamps in postgresql.conf and check the last update time but those are approximate and are not kept for a long time...
I am in the process of migrating a SQL 2008 R2 database between software versions (6 years old to current schema.) There are a few auditing tables with SQL TimeStamp columns on them. Am doing this by copying data out of original tables into the new structure - the change is fairly complex as you might expect after 6 years.
Is there a way to preserve the fingerprint of the timestamps as I move it into a new database or a best practise way of keeping the audit traceability of this data?
Thanks
You can convert a timestamp to varbinary(8) to preserve it:
select cast([timestamp] as varbinary(8))
But the value of timestamp itself is not particularly useful: it does not translate to a particular time. In the future, MSDN suggests it might be renamed to the more appropriate rowversion.
I have used SQL 2014 and it allows rowversion, for your case, you have to convert the timestamp ie. to varbinary and save in another table for reference in future. Timestamp is a read only column.
This consists of two questions:
Is MySQL's timestamp field really faster than datetime field in "order by" query?
If the answer to above question is yes, how much faster it could be? Supposed in a table of 100 million rows and frequently sort a bunch of 100-200k rows based on timestamp field inside MySQL, will the sort time improvement be offseted by converting timestamp to readable string format in outside program?
The easiest way to find out is to write a unit test, and actually get some numbers.
My theory was that timestamp would be faster, but according to this blog I am wrong:
http://dbscience.blogspot.com/2008/08/can-timestamp-be-slower-than-datetime.html
This is why I tend to get numbers, by profiling, before I decide where to optimize, as my gut feeling can be quite wrong at times.
So, it may depend on the version of MySQL you are using, but it appears that datetime may be faster.
From what I can tell, the major benefit of using TIMESTAMP over DATETIME is being able to automatically set a value to the current time on row creation and being able to set it (or another column with a little effort) to the current time on row update. This allows for an automatic created and modified date.
Because of the other limitations on the TIMESTAMP column (for example, not being to accept dates outside of a certain range or changing if the server time zone changes), DATETIME is going to be preferable if you're not needing one of the two features from above.
However, if you're merely interested in storing a Unix timestamp in the database and not needing the database to ever convert it, you could probably store it as an unsigned integer directly and achieve slightly performance.
In the recent project, we had an issue with the performance of few queries that relied heavily on ordering the results by datetime field (MSSQL 2008 database).
When we executed the queries with ORDER BY RecordDate DESC (or ASC) the queries executed 10x slower than without that. Ordering by any other field didn't produce such slow results.
We tried all the indexing options, used the tuning wizard, nothing really made any difference.
One of the suggested solutions was converting the datetime field to the integer field representing the number of seconds or miliseconds in that datetime field. It would be calculated by a simple algorithm, something like "get me the number of seconds from RecordDate to 1980-01-01". This value would be stored at insertion, and the all the sorting would be done on the integer field, and not on the datetime field.
We never tried it, but I'm curious what do you guys think?
I always store dates as ints, using the standardised unix timestamp as most languages I program in use that as a default date-time representation. Obviously, this makes sorting on a date much more efficient.
So, yes, I recommend it :)
I think basically that's how the SQL datetime datatype is stored behind the scenes in SQL Server, so I'd be surprised about these results.
Can you replicate the slowness in Northwinds or Pubs - if so it might be worth a call to MS as it shouldn't be 10x slower. If not then there maybe something odd about your table.
If you are using SQL 2008 and you only need to store dates (not the time portion) you could try using the new date datatype. This has less precision and so should be quicker to sort.
Are the inserts coming from .Net Code...
You could store the DateTime.Ticks value in a bigint column on the DB and index on that.
In terms of updating your existing Database, it should be relatively trivial to write a CLR Function for converting existing DateTimes to TickCount along the lines of
ALTER TABLE dbo.MyTable ADD TickCount BigInt Null
Update dbo.MyTable Set TickCount = CLRFunction(DateTimeColumn)
It definitely feasible and would dramatically improve your sorting abilility
Aren't datetimes stored as a number already?
Do you actually need the DateTime or more specifically, the 'time' part? If not, I would investigate storing the date either as the integer or string representation of an ISO date format (YYYYMMDD) and see if this gives you the require performance boost. Storing ticks/time_t values etc would give you the ability to store the time as well, but I wouldn't really bother with this unless you really need the time component as well. Plus, the added value of storing a humanly readable date is that it is somewhat easier to debug data-related problems simply because you can read and understand the data your program in operating on.
The only sensible way to store dates is as Julian days - unix timestamps are way to short in scope.
By sensible I mean really in the code - it's generally (but not always) better to store dates in the database as datetime.
The database problem that you are experiencing sounds like a different problem. I doubt that changing the field type is going to make a massive difference.
It is hard to be specific without seeing detailed information such as the queries, the amount of records etc, but general advice would be to restructure the order and method of the query to reduce the number of records being ordered - as that can impact massively on performance.
I don't really understand why indexing doesn't help, if SQL behind the covers stores the date as integer representation.
Sorting by the ID columns produces excellent results, or by any other indexed field.
I vote indexing. As I said in the comments above, your dates are stored as two int's behind the scenes anyway (sql 2000 anyway). I can't see this making a difference. Hard to say what the real problem is w/o more info, but my gut feeling is that this isn't the problem. If you have a dev environemnt (and you should :) ), try making the int field there and running the raw queries. It shouldn't be difficult to do, and you'll have conclusive results on that idea.
Is your RecordDate one of the fields in the WHERE clause? Also, is RecordDate your only ORDER BY criteria? Thirdly, is your Query a multi-table join or a single table query? If you are not SELECTING on RecordDate, and using it as the ORDER BY criteria, this could be the cause of the performance issue, as the indexes would not really contribute to the sort in this case. The indexes would try to solve the join issues, and then the sort would happen afterwards.
If this is the case, then changing the data-type of your RecordDate may not help you much, as you are still applying a sort on a recordset after the fact.
I've seen a BI database where the dates are stored as an integer in YYYMMDD format. A separate table is used to relate these ints to the equivalent datetime, formatted string, year number, quarter number, month number, day of week, holiday status, etc. All you have to do is join to that table to get anything date related that you need. Very handy.
I would advise you to use a Julian date as used in Excel (link text). All financial applications are using this representation to gain performance and it provides a relatively good range of values.
SELECT CAST(REPLACE(convert(varchar, GETDATE(), 102),'.','')AS INT)
-- works quite well (and quick!).
I believe the datetime is physically stored as float so the improvement would be the same as when converting float to INT.
I would rather use indexes as that is what they are designed for, and the datatime is designed for storing dates with times. There is a set of functions associated with the datetime so if you decide to use custom storage type you will need to take care of that yourself.