Related
Say I have the following schema:
SENSOR
--------------
ID (numeric)
READ_DATE (date)
VALUE (numeric)
I want to find spikes in data that lasts at least X amount of days. We take 1 reading from the sensor only once per day so ID and READ_DATE are pretty much interchangeable in terms of uniqueness.
For example I have the following records:
1, 2019-01-01, 100
2, 2019-01-02, 1000
3, 2019-01-03, 1500
4, 2019-01-04, 1100
5, 2019-01-05, 500
6, 2019-01-06, 700
7, 2019-01-07, 1500
8, 2019-01-08, 2000
In this example, for X = 2 with VALUE >= 1000, I want to get row 3, 4, 8 because (2, 3), (3, 4), (7, 8) are consecutively >= to 1000.
I am not sure about how to approach this. I was thinking of doing a COUNT window function but don't know how to check whether there are X records >= 1000.
This is about as generic as I think this can get.
First I create some data, using a table variable, but this could be a temporary/ physical table:
DECLARE #table TABLE (id INT, [date] DATE, [value] INT);
INSERT INTO #table SELECT 1, '20190101', 100;
INSERT INTO #table SELECT 2, '20190102', 1000;
INSERT INTO #table SELECT 3, '20190103', 1500;
INSERT INTO #table SELECT 4, '20190104', 1100;
INSERT INTO #table SELECT 5, '20190105', 500;
INSERT INTO #table SELECT 6, '20190106', 700;
INSERT INTO #table SELECT 7, '20190107', 1500;
INSERT INTO #table SELECT 8, '20190108', 2000;
Then I use a CTE (which could be swapped out for a less efficient subquery):
WITH x AS (
SELECT
*,
CASE WHEN [value] >= 1000 THEN 1 END AS spike
FROM
#table)
SELECT
x2.id,
x2.[date],
x2.[value]
FROM
x x1
INNER JOIN x x2 ON x2.id = x1.id + 1
WHERE
x1.spike = 1
AND x2.spike = 1;
This assumes your ids are sequential, if they aren't you would need to join on date instead, which is trickier.
Results:
id date value
3 2019-01-03 1500
4 2019-01-04 1100
8 2019-01-08 2000
Okay, this isn't Postgres, and it isn't very generic (recursive CTE), but it seems to work??
DECLARE #spike_length INT = 3;
WITH x AS (
SELECT
*,
CASE WHEN [value] >= 1000 THEN 1 ELSE 0 END AS spike
FROM
#table),
y AS (
SELECT
x.id,
x.[date],
x.[value],
x.spike AS spike_length
FROM
x
WHERE
id = 1
UNION ALL
SELECT
x.id,
x.[date],
x.[value],
CASE WHEN x.spike = 0 THEN 0 ELSE y.spike_length + 1 END
FROM
y
INNER JOIN x ON x.id = y.id + 1)
SELECT * FROM y WHERE spike_length >= #spike_length;
Results:
id date value spike_length
4 2019-01-04 1100 3
You can approach this as a gaps-and-islands problem -- finding consecutive values above the threshold. The following gets the first date of such sequences:
select s.read_date
from (select s.*,
row_number() over (order by date) as seqnum
from sensor s
where value >= 1000
) s
group by (date - seqnum * interval '1 day')
having count(*) >= 2;
The observation here is that (date - seqnum * interval '1 day') is constant for rows that are adjacent.
You can get the original rows with one more layer of subqueries:
select s.*
from (select s.*, count(*) over (partition by (date - seqnum * interval '1 day') as cnt
from (select s.*,
row_number() over (order by date) as seqnum
from sensor s
where value >= 1000
) s
) s
where cnt >= 2;
I ended up with the following:
-- this parts helps filtering values < 1000 later on
with a as (
select *,
case when value >= 1000 then 1 else 0 end as indicator
from sensor),
-- using the indicator, create a window that calculates the length of the spike
b as (
select *,
sum(indicator) over (order by id asc rows between 2 preceding and current row) as spike
from a)
-- now filter out all spikes < 3
-- (because the window has a size of 3, it can never be larger than 3, so = 3 is okay)
select id, value from b where spike = 3;
This is expanding on #Gordon Linoff's answer, but which I found too complicated.
If you are able to use analytic functions, then you should be able to do something like this to get what you need (I altered your 1000 limit to 1500 else it would have brought back all rows which consecutively add up to 1000 and above)
CREATE TABLE test1 (
id number,
value number
);
insert all
into test1 (id, value) values (1, 100)
into test1 (id, value) values (2, 1000)
into test1 (id, value) values (3, 1500)
into test1 (id, value) values (4, 1100)
into test1 (id, value) values (5, 500)
into test1 (id, value) values (6, 700)
into test1 (id, value) values (7, 1500)
into test1 (id, value) values (8, 2000)
select * from dual;
EDIT - After re-reading again - and from comment - have re-done to answer the actual question! Using 2 lags - one to make sure previous day was 1000 or greater and another to count up how many times has happened for X filtering.
SELECT * FROM
(
SELECT id,
value,
spike,
CASE WHEN spike = 0 THEN 0 ELSE (spike + LAG(spike, 1, 0) OVER (ORDER BY id) + 1) END as SPIKE_LENGTH
FROM (
select id,
value,
CASE WHEN LAG(value, 1, 0) OVER (ORDER BY id) >= 1000 AND value >= 1000 THEN 1 ELSE 0 END AS SPIKE
from test1
)
)
WHERE spike_length >= 2;
Which returns
ID Value spike spike_length
3 1500 1 2
4 1100 1 3
8 2000 1 2
If you increase the spike length filter to >= 3 - only get ID 4 which is the only ID with 3 over 1000 in a row.
I have three columns
suppose
row no column1 column2 column3
1 A B C
2 A B C
3 D E F
4 G H I
5 G H C
I want to generate code by combining these three column values
For Eg.
1)ABC001
2)ABC002
3)DEF001
4)GHI001
5)GHC001
by checking combination of three columns
logic is that
if values of three columns are same then like first time it shows 'ABC001'
and 2nd time it shows 'ABC002'
You can try this:
I dont know what you want for logic with 00, but you can add them manuel or let the rn decide for you
declare #mytable table (rowno int,col1 nvarchar(50),col2 nvarchar(50),col3 nvarchar(50)
)
insert into #mytable
values
(1,'A', 'B', 'C'),
(2,'A', 'B', 'C'),
(3,'D', 'E', 'F'),
(4,'G', 'H', 'I'),
(5,'G', 'H', 'C')
Select rowno,col1,col2,col3,
case when rn >= 10 and rn < 100 then concatcol+'0'+cast(rn as nvarchar(50))
when rn >= 100 then concatcol+cast(rn as nvarchar(50))
else concatcol+'00'+cast(rn as nvarchar(50)) end as ConcatCol from (
select rowno,col1,col2,col3
,Col1+col2+col3 as ConcatCol,ROW_NUMBER() over(partition by col1,col2,col3 order by rowno) as rn from #mytable
) x
order by rowno
My case when makes sure when you hit number 10 it writes ABC010 and when it hits above 100 it writes ABC100 else if its under 10 it writes ABC001 and so on.
Result
TSQL: CONCAT(column1,column2,column3,RIGHT(REPLICATE("0", 3) + LEFT(row_no, 3), 3))
You should combine your columns like below :
SELECT CONVERT(VARCHAR(MAX), ROW_NUMBER() OVER(ORDER BY
(
SELECT NULL
)))+') '+DATA AS Data
FROM
(
SELECT column1+column2+column3+'00'+CONVERT(VARCHAR(MAX), ROW_NUMBER() OVER(PARTITION BY column1,
column2,
column3 ORDER BY
(
SELECT NULL
))) DATA
FROM <table_name>
) T;
Result :
1)ABC001
2)ABC002
3)DEF001
4)GHI001
5)GHC001
MySQL:
CONCAT(column1,column2,column3,LPAD(row_no, 3, '0'))
[you will need to enclose the 'row no' in ticks if there is a space in the name of the field instead of underscore.]
i have following table "vehicle_data" :
ID ALERT_TYPE VALUE
58 2 1
58 1 1
104 1 1
104 2 1
Here alert_type = 2 is for GPS value and alert_type=1 is for engine_value .
so if alert_type=2 and its value is =1 then it means its value is correct.
when alert_type=2 and its value is =0 then it means its value is wrong.
same for alert_type=1
so now here i want the following output:
ID gps engine_value
58 1 1
104 1 1
how can i perform this query??
You can do it like this.
SELECT ID
,CASE WHEN [ALERT_TYPE]=2 and [value ]=1 THEN 1 ELSE 0 END as gps
,CASE WHEN [ALERT_TYPE]=1 and [value ]=1 THEN 1 ELSE 0 END as engine
FROM vehicle_data
SELECT ID, alert_type=2 AS gps, alert_type=1 AS [engine] FROM vehicle_data WHERE value=1;
EDITED to account for your explanation of VALUE.
Schema
CREATE TABLE table3 (id int, ALERT_TYPE int)
INSERT table3 VALUES (58, 1), (58, 2), (104, 1), (104, 2)
Query
SELECT *
FROM (
SELECT ID
,ROW_NUMBER() OVER (
PARTITION BY id ORDER BY id
) AS row_num
,gps = CASE
WHEN ALERT_TYPE = 2
THEN 1
ELSE 0
END
,engine = CASE
WHEN ALERT_TYPE = 1
THEN 1
ELSE 0
END
FROM table3
) a
WHERE a.row_num = 1
Output
ID gps engine
58 1 0
104 0 1
One possible way using subqueries :
select
Ids.ID
, gps.VALUE 'gps'
, engine_value.VALUE 'engine_value'
from (select distinct ID from vehicle_data) Ids
left join
(select ID, VALUE from vehicle_data where ALERT_TYPE = 2) gps
on gps.ID = Ids.ID
left join
(select ID, VALUE from vehicle_data where ALERT_TYPE = 1) engine_value
on engine_value.ID = Ids.ID
[SQL Fiddle demo]
I hope this should work for you,
Select ID,sum(gps) as gps ,sum(engine) as engine from
(SELECT ID
,CASE WHEN [ALERT_TYPE]=2 and [value ]=1 THEN 1 ELSE 0 END as gps
,CASE WHEN [ALERT_TYPE]=1 and [value ]=1 THEN 1 ELSE 0 END as engine
FROM vehicle_data
)a
group by id
select x.id,x.alert_type as GPS,x.value as engine_value from (
select ID,alert_type,value,ROW_NUMBER() over (partition by id order by alert_type ) as Rnk from mytable
)x
where Rnk=1
Please check this query in SQL :
create table mytable (id int, alert_type int, value int);
insert into mytable (id, alert_type, value)
values (58, 2, 1),
(58, 1, 1),
(104, 1, 1),
(104, 2, 1);
SELECT distinct ID
,(select count (id) from mytable mt where mt.id=mytable.id and mt.[ALERT_TYPE]=2 and mt.[value ]=1) as gps
,(select count (id) from mytable mt where mt.id=mytable.id and mt.[ALERT_TYPE]=1 and mt.[value ]=1) as engine
FROM mytable
BASED ON YOUR QUESTION I BELIEVE YOU WANT THE DATA IN COLUMN AND TO SUIT YOUR REQUIREMENT I HAVE MADE A SQL FIDDLE WORKING - CODE IS ALSO MENTIONED BELOW -
HERE YOU GO WITH THE WORKING FIDDLE -
WORKING DEMO
SQL CODE FOR REFERNECE -
CREATE TABLE ALERTS (ID INT, ALERT_TYPE INT, VALUE INT)
INSERT INTO ALERTS VALUES (58,2,1)
INSERT INTO ALERTS VALUES (58,1,0)
INSERT INTO ALERTS VALUES (104,1,1)
INSERT INTO ALERTS VALUES (104,2,0)
CREATE TABLE ALERTSVALUE (ID INT, gps INT,engine INT)
INSERT INTO ALERTSVALUE VALUES (58,1,0)
INSERT INTO ALERTSVALUE VALUES (104,0,1)
SELECT A.ID,
CASE A.ALERT_TYPE WHEN 2 THEN 1 ELSE 0 END AS GPS,
CASE A.ALERT_TYPE WHEN 1 THEN 1 ELSE 0 END AS ENGINE_VALUE,
A.VALUE FROM ALERTS A WHERE A.VALUE = 1
EDIT BASED ON COMMENT - TO MERGE THE ROWS FOR BOTH GPS AND ENGINE_VALUE:
SELECT X.ID,X.ALERT_TYPE as GPS,X.VALUE as ENGINE_VALUE
FROM (
SELECT ID,ALERT_TYPE ,VALUE ,ROW_NUMBER() OVER (PARTITION BY ID ORDER BY alert_type ) AS [Rank] FROM ALERTS
)X
WHERE [Rank]=1
SQL FIDDLE DEMO
i have a table "mydata" with some data data :
id name position
===========================
4 foo -3
6 bar -2
1 baz -1
3 knork -1
5 lift 0
2 pitcher 0
i fetch the table ordered using order by position ASC;
the position column value may be non unique (for some reason not described here :-) and is used to provide a custom order during SELECT.
what i want to do :
i want to normalize the table column "position" by associating a unique position to each row which doesnt destroy the order. furthermore the highest position after normalising should be -1.
wished resulting table contents :
id name position
===========================
4 foo -6
6 bar -5
1 baz -4
3 knork -3
5 lift -2
2 pitcher -1
i tried several ways but failed to implement the correct update statement.
i guess that using
generate_series( -(select count(*) from mydata), -1)
is a good starting point to get the new values for the position column but i have no clue how to merge that generated column data into the update statement.
hope somebody can help me out :-)
Something like:
with renumber as (
select id,
-1 * row_number() over (order by position desc, id) as rn
from foo
)
update foo
set position = r.rn
from renumber r
where foo.id = r.id
and position <> r.rn;
SQLFiddle Demo
Try this one -
Query:
CREATE TABLE temp
(
id INT
, name VARCHAR(10)
, position INT
)
INSERT INTO temp (id, name, position)
VALUES
(4, 'foo', -3),
(6, 'bar', -2),
(1, 'baz', -1),
(3, 'knork', -1),
(5, 'lift', 0),
(2, 'pitcher', 0)
SELECT
id
, name
, position = -ROW_NUMBER() OVER (ORDER BY position DESC, id)
FROM temp
ORDER BY position
Update:
UPDATE temp
SET position = t.rn
FROM (
SELECT id, rn = - ROW_NUMBER() OVER (ORDER BY position DESC, id)
FROM temp
) t
WHERE temp.id = t.id
Output:
id name position
----------- ---------- --------------------
4 foo -6
6 bar -5
3 knork -4
1 baz -3
5 lift -2
2 pitcher -1
#a_horse_with_no_name is really near the truth - thank you !
UPDATE temp
SET position=t.rn
FROM (SELECT
id, name,
-((select count( *)
FROM temp)
+1-row_number() OVER (ORDER BY position ASC)) as rn
FROM temp) t
WHERE temp.id=t.id;
SELECT * FROM temp ORDER BY position ASC;
see http://sqlfiddle.com/#!1/d1770/6
update mydata temp1, (select a.*,#var:=#var-1 sno from mydata a, (select #var:=0) b
order by position desc, id asc) temp2
set temp1.position = temp2.sno
where temp1.id = temp2.id;
I have a (in my oppinion) tricky SQL problem.
I got a table with subscriptions. Each subscription has an ID and a set of attributes which will change over time. When an attribute value changes a new row is created with the subscription key and the new values – but ONLY for the changed attributes. The values for the attributes that weren’t changed are left empty. It looks something like this (I left out the ValidTo and ValidFrom dates that I use to sort the result correctly):
SubID Att1 Att2
1 J
1 L
1 B
1 H
1 A H
I need to transform this table so I can get the following result:
SubID Att1 Att2
1 J
1 J L
1 B L
1 B H
1 A H
So basically; if an attribute is empty then take the previous value for that attribute.
Anything solution goes…. I mean it doesn’t matter what I have to do to get the result: a view on top of the table, an SSIS package to create a new table or something third.
You can do this with a correlated subquery:
select t.subid,
(select t2.att1 from t t2 where t2.rowid <= t.rowid and t2.att1 is not null order by rowid desc limit 1) as att1,
(select t2.att2 from t t2 where t2.rowid <= t.rowid and t2.att2 is not null order by rowid desc limit 1) as att1
from t
This assumes that you have a rowid or equivalent (such as date time created) that specifies the ordering of the rows. It also uses limit to limit the results. In other databases, this might use top instead. (And Oracle uses a slightly more complex expression.)
I would write this using ValidTo. However, because there is ValidTo and ValidFrom, the actual expression is much more complicated. I would need for the question to clarify the rules for using these values with respect to imputing values at other times.
this one works in oracle 11g
select SUBID
,NVL(ATT1,LAG(ATT1) over(order by ValidTo)) ATT1
,NVL(ATT2,lag(ATT2) over(order by ValidTo)) ATT2
from table_name
i agree with Gordon Linoff and Jack Douglas.this code has limitation as when multiple records with nulls are inserted..
but below code will handle that..
select SUBID
,NVL(ATT1,LAG(ATT1 ignore nulls) over(order by VALIDTO)) ATT1
,NVL(ATT2,LAG(ATT2 ignore nulls) over(order by VALIDTO)) ATT2
from Table_name
please see sql fiddle
http://sqlfiddle.com/#!4/3b530/4
Assuming (based on the fact that you mentioned SSIS) you can use OUTER APPLY to get the previous row:
DECLARE #T TABLE (SubID INT, Att1 CHAR(1), Att2 CHAR(2), ValidFrom DATETIME);
INSERT #T VALUES
(1, 'J', '', '20121201'),
(1, '', 'l', '20121202'),
(1, 'B', '', '20121203'),
(1, '', 'H', '20121204'),
(1, 'A', 'H', '20121205');
SELECT T.SubID,
Att1 = COALESCE(NULLIF(T.att1, ''), prev.Att1, ''),
Att2 = COALESCE(NULLIF(T.att2, ''), prev.Att2, '')
FROM #T T
OUTER APPLY
( SELECT TOP 1 Att1, Att2
FROM #T prev
WHERE prev.SubID = T.SubID
AND prev.ValidFrom < t.ValidFrom
ORDER BY ValidFrom DESC
) prev
ORDER BY T.ValidFrom;
(I've had to add random values for ValidFrom to ensure the order by is correct)
EDIT
The above won't work if you have multiple consecutive rows with blank values - e.g.
DECLARE #T TABLE (SubID INT, Att1 CHAR(1), Att2 CHAR(2), ValidFrom DATETIME);
INSERT #T VALUES
(1, 'J', '', '20121201'),
(1, '', 'l', '20121202'),
(1, 'B', '', '20121203'),
(1, '', 'H', '20121204'),
(1, '', 'J', '20121205'),
(1, 'A', 'H', '20121206');
If this is likely to happen you will need two OUTER APPLYs:
SELECT T.SubID,
Att1 = COALESCE(NULLIF(T.att1, ''), prevAtt1.Att1, ''),
Att2 = COALESCE(NULLIF(T.att2, ''), prevAtt2.Att2, '')
FROM #T T
OUTER APPLY
( SELECT TOP 1 Att1
FROM #T prev
WHERE prev.SubID = T.SubID
AND prev.ValidFrom < t.ValidFrom
AND COALESCE(prev.Att1 , '') != ''
ORDER BY ValidFrom DESC
) prevAtt1
OUTER APPLY
( SELECT TOP 1 Att2
FROM #T prev
WHERE prev.SubID = T.SubID
AND prev.ValidFrom < t.ValidFrom
AND COALESCE(prev.Att2 , '') != ''
ORDER BY ValidFrom DESC
) prevAtt2
ORDER BY T.ValidFrom;
However, since each OUTER APPLY is only returning one value I would change this to a correlated subquery, since the above will evaluate PrevAtt1.Att1 and `PrevAtt2.Att2' for every row whether required or not. However if you change this to:
SELECT T.SubID,
Att1 = COALESCE(
NULLIF(T.att1, ''),
( SELECT TOP 1 Att1
FROM #T prev
WHERE prev.SubID = T.SubID
AND prev.ValidFrom < t.ValidFrom
AND COALESCE(prev.Att1 , '') != ''
ORDER BY ValidFrom DESC
), ''),
Att2 = COALESCE(
NULLIF(T.att2, ''),
( SELECT TOP 1 Att2
FROM #T prev
WHERE prev.SubID = T.SubID
AND prev.ValidFrom < t.ValidFrom
AND COALESCE(prev.Att2 , '') != ''
ORDER BY ValidFrom DESC
), '')
FROM #T T
ORDER BY T.ValidFrom;
The subquery will only evaluate when required (ie. when Att1 or Att2 is blank) rather than for every row. The execution plan does not show this, and in fact the "Actual Execution Plan" of the latter appears more intensive it almost certainly won't be. But as always, the key is testing, run both on your data and see which performs the best, and check the IO statistics for reads etc.
I never touched SQL Server, but I read that it supports analytical functions just like Oracle.
> select * from MYTABLE order by ValidFrom;
SUBID A A VALIDFROM
---------- - - -------------------
1 J 2012-12-06 15:14:51
2 j 2012-12-06 15:15:20
1 L 2012-12-06 15:15:31
2 l 2012-12-06 15:15:39
1 B 2012-12-06 15:15:48
2 b 2012-12-06 15:15:55
1 H 2012-12-06 15:16:03
2 h 2012-12-06 15:16:09
1 A H 2012-12-06 15:16:20
2 a h 2012-12-06 15:16:29
select
t.SubID
,last_value(t.Att1 ignore nulls)over(partition by t.SubID order by t.ValidFrom rows between unbounded preceding and current row) as Att1
,last_value(t.Att2 ignore nulls)over(partition by t.SubID order by t.ValidFrom rows between unbounded preceding and current row) as Att2
,t.ValidFrom
from MYTABLE t;
SUBID A A VALIDFROM
---------- - - -------------------
1 J 2012-12-06 15:45:33
1 J L 2012-12-06 15:45:41
1 B L 2012-12-06 15:45:49
1 B H 2012-12-06 15:45:58
1 A H 2012-12-06 15:46:06
2 j 2012-12-06 15:45:38
2 j l 2012-12-06 15:45:44
2 b l 2012-12-06 15:45:53
2 b h 2012-12-06 15:46:02
2 a h 2012-12-06 15:46:09
with Tricky1 as (
Select SubID, Att1, Att2, row_number() over(order by ValidFrom) As rownum
From Tricky
)
select T1.SubID, T1.Att1, T2.Att2
from Tricky1 T1
cross join Tricky1 T2
where (ABS(T1.rownum-T2.rownum) = 1 or (T1.rownum = 1 and T2.rownum = 1))
and T1.Att1 is not null
;
Also, have a look at accessing previous value, when SQL has no notion of previous value, here.
I was at it for quite a while now. I found a rather simple way of doing it. Not the best solution as such as i know there must be other way, but here it goes.
I had to consolidates duplicates too and in 2008R2.
So if you can try to create a table which contains one set of duplicates records.
According to your example create one table where 'ATT1' is blank. Then use Update queries with Inner join on 'SubId' to populate the data that you need