How to determine when a time stamp does not exist in a table - sql

I have a table that receives data on an hourly basis. Part of this import process writes the timestamp of the import to the table. My question is, how can I build a query to produce a result set of the periods of time when the import did not write to the table?
My first thought is to have a table of static int and just do an outer join and look for nulls on the right side, but this seems kind of sloppy. Is there a more dynamic way to produce a result set for the times the import failed based on the timestamp?
This is a MS SQL 2000 box.
Update: I think I've got it. The two answers already provided are great, but instead what I'm working on is a function that returns a table of the values I am looking for for a given time frame. Once I get it finished I'll post the solution here.

Here's a slightly modified solution from this article in my blog:
Flattening timespans: SQL Server
DECLARE #t TABLE
(
q_start DATETIME NOT NULL,
q_end DATETIME NOT NULL
)
DECLARE #qs DATETIME
DECLARE #qe DATETIME
DECLARE #ms DATETIME
DECLARE #me DATETIME
DECLARE cr_span CURSOR FAST_FORWARD
FOR
SELECT s_timestamp AS q_start,
DATEADD(minute, 1, s_timestamp) AS q_end
FROM [20090611_timespans].t_span
ORDER BY
q_start
OPEN cr_span
FETCH NEXT
FROM cr_span
INTO #qs, #qe
SET #ms = #qs
SET #me = #qe
WHILE ##FETCH_STATUS = 0
BEGIN
FETCH NEXT
FROM cr_span
INTO #qs, #qe
IF #qs > #me
BEGIN
INSERT
INTO #t
VALUES (#ms, #me)
SET #ms = #qs
END
SET #me = CASE WHEN #qe > #me THEN #qe ELSE #me END
END
IF #ms IS NOT NULL
BEGIN
INSERT
INTO #t
VALUES (#ms, #me)
END
CLOSE cr_span
This will return you the consecutive ranges when updates did happen (with a minute resolution).
If you have an index on your timestamp field, you may issue the following query:
SELECT *
FROM records ro
WHERE NOT EXISTS
(
SELECT NULL
FROM records ri
WHERE ri.timestamp >= DATEADD(minute, -1, ro.timestamp)
AND ri.timestamp < ro.timestamp
)

I was thinking something like this:
select 'Start' MissingStatus, o1.LastUpdate MissingStart
from Orders o1
left join Orders o2
on o1.LastUpdate between
dateadd(ss,1,o2.LastUpdate) and dateadd(hh,1,o2.LastUpdate)
where o2.LastUpdate is null
union all
select 'End', o1.LastUpdate MissingEnd
from Orders o1
left join Orders o2
on o1.LastUpdate between
dateadd(hh,-1,o2.LastUpdate) and dateadd(ss,-1,o2.LastUpdate)
where o2.LastUpdate is null
order by 2

Related

Using SQL Cursor, To find the date > 45 days difference from initial date, then again from 2nd date found, again > 45 and find next date

Declare #quoteReference NVARCHAR(20)
Declare #policyNo NVARCHAR(20)
Declare #todaysDate DATE
Declare #time int
Declare #ddiff int
Declare #cnt varchar(10)
Declare EmpCursor Cursor for
select quote_reference, Miscellaneous_TodaysDate, diff from #temp1
OPEN EmpCursor
FETCH NEXT FROM EmpCursor INTO #quoteReference ,#todaysDate,#ddiff WHILE(##FETCH_STATUS = 0)
BEGIN
select #quoteReference = quote_reference
--#policyno = policy_policyNo,
-- #time = Miscellaneous_CurrentTime
from #temp1 where
Miscellaneous_TodaysDate = #todaysDate and diff > 45
if
#ddiff < 45
update #temp1
set counts = 'N'
where Miscellaneous_TodaysDate = #todaysDate
else
update #temp1
set counts = 'Y'
where Miscellaneous_TodaysDate = #todaysDate
FETCH NEXT FROM EmpCursor INTO #quoteReference, #policyNo, #todaysDate
END
CLOSE EmpCursor
DEALLOCATE EmpCursor
The first time quote_reference entered in the database is 31/05/2021.
I need another date where the same record came again in the database after more than 45 days (so in this case its 64 days i.e 03/08/2021) then 03/08/2021 date becomes my 2nd initial date and from that date same rules and conditions, i.e. to find date where the same record
entered again after more than 45 days difference.
DATA:
have you tried using the LAG function. link here - https://learn.microsoft.com/en-us/sql/t-sql/functions/lag-transact-sql?view=sql-server-ver16
I dont remember when was the last time I used cursor, you can very much achieve what you want in a SQL statement

How to update a table using while loops and waitfor delay to insert current date with a one second delay between records?

I'd like to create a table that contains two columns (id int, today datetime) and, using while loops, to insert the current date every 1 second. However, the resulting table shows the same time for all rows. Below is my code. Can anyone help me understand what I'm doing wrong, please? Thank you!
declare #mytable table (id int, today datetime)
declare #id int=1
declare #today datetime=getdate()
while #id<10
begin
waitfor delay '00:00:01'
insert into #mytable values (#id,#today)
set #id=#id+1
end
The reason every row has the same value is because you aren't setting the value of #Today anywhere apart from before your WHILE loop. GETDATE() returns a scalar value, and setting a variable to that value means it will be set the value that GETDATE() returned at the time the SET was run. The value of the variable won't change after time has passed. For example:
DECLARE #d datetime;
SET #d = GETDATE();
SELECT #d, GETDATE(); --Will return very similar values
WAITFOR DELAY '00:00:05';
SELECT #d, GETDATE(); --#d will have the same value as before, as its value is static, but GETDATE()'s value will have changed.
To do what you're after, I don't see any need for the variable for #Today, this would work fine:
DECLARE #mytable table (id int,
today datetime);
DECLARE #id int = 1;
WHILE #id < 10
BEGIN
WAITFOR DELAY '00:00:01';
INSERT INTO #mytable
VALUES (#id, GETDATE());
SET #id = #id + 1;
END;
However a loop is a bad choice anyway, as an RDBMS excels at set based operations, not iterative. You would be far better to achieve what you're after by doing:
DECLARE #mytable table (id int,
today datetime);
DECLARE #id int = 1;
WITH N AS (
SELECT N
FROM (VALUES(NULL),(NULL),(NULL),(NULL),(NULL),(NULL),(NULL),(NULL),(NULL),(NULL)) N(N)),
Tally AS(
SELECT ROW_NUMBER() OVER (ORDER BY (SELECT NULL)) -1 AS I
FROM N N1
CROSS JOIN N N2 --Not actually eneded here, but shows how to increase row count
)
INSERT INTO #mytable (id,
today)
SELECT TOP 10
T.I + #ID,
DATEADD(SECOND, T.I, GETDATE())
FROM Tally T
ORDER BY T.I;
This builds an inline tally table, and then inserts a value for a row for 10 ID, and adds 1 second to each incremented ID.

Better alternative for Cursor using While loop to iterate Sql

Good Day,
I am trying to fetch dates from one of my table, containing records in millions, in database and save them in a variable using Cursor. After fetching dates i am inserting records in db at that particular date. For this I am using While Loop. It turns out that While loop is really slowing down the performance, it is taking about hours to complete execution. I am including a part of a query for further clearance.
declare #tranDateCursor cursor;
declare #today as date;
begin
set #tranDateCursor = cursor
for
select distinct transferDate
from transactions
where [type] = 'customer'
open #tranDateCursor
fetch next from #tranDateCursor
into #today
while ##fetch_status = 0
begin
declare #yesterday as date
set #yesterday = (
select top(1) transferDate
from (
select distinct(transferDate) AS transferDate
from transactions
where transferDate < #today
and [type] = 'customer'
) as ODC_DATES
order by ODC_DATES.transferDate desc
)
insert into transactions([type],transferDate)
select 'customer'
,#today
from transactions xt
right outer join x_itransaction as y
on y.customer_account = xt.customer_account
AND y.transferDate = #yesterday
AND xt.transferDate = #today
where xt.transactionId is null
and y.transferDate = #yesterday
and y.[type] ='customer'
end
end
I have tried using Table Variables instead of CURSOR with WHILE loop, but it turns out that it was too taking very much time to run. My concern is that, Is there a better alternative for a while loop in this particular scenario?

Lock entire table stored procedure

Guys I have a stored procedure that inserts a new value in the table, only when the last inserted value is different.
CREATE PROCEDURE [dbo].[PutData]
#date datetime,
#value float
AS
IF NOT EXISTS(SELECT * FROM Sensor1 WHERE SensorTime <= #date AND SensorTime = (SELECT MAX(SensorTime) FROM Sensor1) AND SensorValue = #value)
INSERT INTO Sensor1 (SensorTime, SensorValue) VALUES (#date, #value)
RETURN 0
Now, since I'm doing this at a high frequency (say every 10ms), the IF NOT EXISTS (SELECT) statement is often getting old data, and because of this I'm getting duplicate data. Would it be possible to lock the entire table during the stored procedure, to make sure the SELECT statement always receives the latest data?
According to the poster's comments to the question, c# code receives a value from a sensor. The code is supposed to insert the value only if it is different from the previous value.
Rather than solving this in the database, why not have the code store the last value inserted and only invoke the procedure if the new value is different? Then the procedure will not need to check whether the value exists in the database; it can simply insert. This will be much more efficient.
You could write it like this :
CREATE PROCEDURE [dbo].[PutData]
#date datetime,
#value float
AS
BEGIN TRANSACTION
INSERT INTO Sensor1 (SensorTime, SensorValue)
SELECT SensorTime = #date,
SensorValue = #value
WHERE NOT EXISTS(SELECT *
FROM Sensor1 WITH (UPDLOCK, HOLDLOCK)
WHERE SensorValue = #value
AND SensorTime <= #date
AND SensorTime = (SELECT MAX(SensorTime) FROM Sensor1) )
COMMIT TRANSACTION
RETURN 0
Thinking a bit about it, you could probably write it like this too:
CREATE PROCEDURE [dbo].[PutData]
#date datetime,
#value float
AS
BEGIN TRANSACTION
INSERT INTO Sensor1 (SensorTime, SensorValue)
SELECT SensorTime = #date,
SensorValue = #value
FROM (SELECT TOP 1 SensorValue, SensorTime
FROM Sensor1 WITH (UPDLOCK, HOLDLOCK)
ORDER BY SensorTime DESC) last_value
WHERE last_value.SensorTime <= #date
AND last_value.SensorValue <> #value
COMMIT TRANSACTION
RETURN 0
Assuming you have a unique index (PK?) on SensorTime this should be quite fast actually.

SQL query with start and end dates - what is the best option?

I am using MS SQL Server 2005 at work to build a database. I have been told that most tables will hold 1,000,000 to 500,000,000 rows of data in the near future after it is built... I have not worked with datasets this large. Most of the time I don't even know what I should be considering to figure out what the best answer might be for ways to set up schema, queries, stuff.
So... I need to know the start and end dates for something and a value that is associated with in ID during that time frame. SO... we can the table up two different ways:
create table xxx_test2 (id int identity(1,1), groupid int, dt datetime, i int)
create table xxx_test2 (id int identity(1,1), groupid int, start_dt datetime, end_dt datetime, i int)
Which is better? How do I define better? I filled the first table with about 100,000 rows of data and it takes about 10-12 seconds to set up in the format of the second table depending on the query...
select y.groupid,
y.dt as [start],
z.dt as [end],
(case when z.dt is null then 1 else 0 end) as latest,
y.i
from #x as y
outer apply (select top 1 *
from #x as x
where x.groupid = y.groupid and
x.dt > y.dt
order by x.dt asc) as z
or
http://consultingblogs.emc.com/jamiethomson/archive/2005/01/10/t-sql-deriving-start-and-end-date-from-a-single-effective-date.aspx
Buuuuut... with the second table.... to insert a new row, I have to go look and see if there is a previous row and then if so update its end date. So... is it a question of performance when retrieving data vs insert/update things? It seems silly to store that end date twice but maybe...... not? What things should I be looking at?
this is what i used to generate my fake data... if you want to play with it for some reason (if you change the maximum of the random number to something higher it will generate the fake stuff a lot faster):
declare #dt datetime
declare #i int
declare #id int
set #id = 1
declare #rowcount int
set #rowcount = 0
declare #numrows int
while (#rowcount<100000)
begin
set #i = 1
set #dt = getdate()
set #numrows = Cast(((5 + 1) - 1) *
Rand() + 1 As tinyint)
while #i<=#numrows
begin
insert into #x values (#id, dateadd(d,#i,#dt), #i)
set #i = #i + 1
end
set #rowcount = #rowcount + #numrows
set #id = #id + 1
print #rowcount
end
For your purposes, I think option 2 is the way to go for table design. This gives you flexibility, and will save you tons of work.
Having the effective date and end date will allow you to have a query that will only return currently effective data by having this in your where clause:
where sysdate between effectivedate and enddate
You can also then use it to join with other tables in a time-sensitive way.
Provided you set up the key properly and provide the right indexes, performance (on this table at least) should not be a problem.
for anyone who can use LEAD Analytic function of SQL Server 2012 (or Oracle, DB2, ...), retrieving data from the 1st table (that uses only 1 date column) would be much much quicker than without this feature:
select
groupid,
dt "start",
lead(dt) over (partition by groupid order by dt) "end",
case when lead(dt) over (partition by groupid order by dt) is null
then 1 else 0 end "latest",
i
from x