'Summing' a date field in SQL - any ideas? - sql

Question

I'm creating an application that is essentially an integrity check between two databases - one is MSSQL and one is an old provider Btrieve. As part of the requirements all columns for every table need to be compared to ensure the data matches. Currently we loop through each table, get the basic count of the table in both DBs, and then delve into the columns. For numeric fields we do a simple SUM, and for text fields we sum up the length of the column for every row. If these match in both DBs, it's a good indicator the data has migrated across correctly.
This all works fine, but I need to develop something similar for datetime fields. Obviously we can't really SUM these fields, so I'm wondering if anyone has ideas on the best way to approach this. I was thinking maybe the seconds since a certain date but the number will be huge.
Any other ideas? Thanks!

Answer 1

The most straightforward answer to me would be to convert the date or datetime fields to integers with the same format. YYYYMMDD or YYYYMMDDHHmmss work just fine as long as your formats use leading zeroes. In SQL Server, you can do something like:
SELECT SUM(CAST(REPLACE(REPLACE(REPLACE(CONVERT(VARCHAR(20),DateTimeColumn,120),' ',''),':',''),'-','') AS BIGINT)) .....
Alternately, you can convert them to either the number of days from a given date ('1970-01-01'), or the number of seconds from a given date ('1970-01-01 00:00:00') if you use time.
SELECT SUM(DATEDIFF(DAY,'19700101',DateColumn)) ....
I'm not familiar enough with Btrieve to know what kinds of functions are available for formatting dates, however.

Answer 2

Using "Except" in SQL on the lines of Numeric fields you can compare the date counts in both the tables. For the Old source you may generate the select statement using excel or in the native database and bring to the SQL Server. For demonstration purpose I have used two tables and showing Except example below.
IF EXISTS (SELECT * FROM sys.objects
WHERE OBJECT_ID = OBJECT_ID(N'[dbo].[DateCompareOld]') AND
TYPE IN (N'U'))
DROP TABLE [dbo].[DateCompareOld]
GO
CREATE TABLE dbo.DateCompareOld
(
AsOf DATETIME
)
INSERT INTO DateCompareOld
SELECT '01/01/2016' UNION ALL
SELECT '01/01/2016' UNION ALL
SELECT '01/01/2016' UNION ALL
SELECT '01/02/2016' UNION ALL
SELECT '01/02/2016' UNION ALL
SELECT '01/02/2016'
IF EXISTS (SELECT * FROM sys.objects WHERE OBJECT_ID = OBJECT_ID(N'[dbo].[DateCompareNew]') AND TYPE IN (N'U'))
DROP TABLE [dbo].[DateCompareNew]
GO
CREATE TABLE dbo.DateCompareNew
(
AsOf DATETIME
)
INSERT INTO DateCompareNew
SELECT '01/01/2016' UNION ALL
SELECT '01/01/2016' UNION ALL
SELECT '01/01/2016' UNION ALL
SELECT '01/02/2016' UNION ALL
SELECT '01/02/2016' UNION ALL
SELECT '01/02/2016'
SELECT AsOf,COUNT(*) AsOfCount
FROM DateCompareOld
GROUP BY AsOf
Except
SELECT AsOf,COUNT(*) AsOfCount
FROM DateCompareNew
GROUP BY AsOf

Answer 3

Unless the date range used by rows in the database is extreme (like dates of astronomical stars being born and dying), it should be just as valid to convert the dates to an integer. This can be done any of several ways and is slightly database-specific, but converting 2016-01-04 to 20,160,104 is going to work fine.
Even SQL Server allows ORD(date_field) like expressions to obtain the internal representation. But this can also be done in a portable, system-agnostic means like
datediff(day, 'January 1, 1901', date_field)
if keeping track of days is sufficient, or
datediff(second, 'January 1, 1901', date_field)
if keeping track of seconds is needed.

Answer 4

Maybe it is not much help, maybe is something:
declare #d1 datetime; set #d1 = '2016-01-05 12:09'
declare #d2 datetime; set #d2 = '1970-04-05 07:09'
declare #d3 datetime; set #d3 = '1999-12-12 23:05'
declare #d4 datetime; set #d4 = '1999-12-12 23:06'
declare #i1 bigint
declare #i2 bigint
declare #i3 bigint
declare #i4 bigint
select #i1 = convert( bigint, convert( timestamp, #d1 ) )
select #i2 = convert( bigint, convert( timestamp, #d2 ) )
select #i3 = convert( bigint, convert( timestamp, #d3 ) )
select #i4 = convert( bigint, convert( timestamp, #d4 ) )
select #i1
select #i2
select #i3
select #i4
select #i1 ^ #i2 ^ #i3 ^ #i4

Answer 5

I think you could do something like this on the SQL Server side to find the center ("average") value of the column. Then use that value on the Btrieve side to avoid overflow issues where I'm guessing you're more constrained.
-- January 1, 2000 value pulled out of the air as a stab in the dark
select
dataadd(
second,
avg(cast(datediff(datediff(second, '20000101', <data>) as bigint)),
'20000101'
) /* find the center */
I wouldn't be surprised if you had to resort to a floating point type with Btrieve or partition your scans into smaller ranges to avoid intermediate sums that get too big. And you might want to use a cursor and randomize the ordering of the rows so you don't hit them in a sorted order that causes an overflow. At this point I'm just speculating since I haven't seen any of the data and my knowledge of Btrieve is so ancient and minimal to begin with.
I also have a feeling that some of this effort is all about satisfying some uneasiness on the part of non-technical stakeholders. I'm sure you could come up with checksums and hashes that would work better but this summing concept is the one they can grasp and will set their minds at ease on top of being somewhat easier to implement quickly.