SQL - Average every n rows for items with same ID - sql

I found a lot of similart posts, but any of them answered what I needed. Im trying to sumarize a big table (200M rows). What I need is to get the average every n (5 or something like that) for each ID. I've been trying with this:
select id, dev_id,
row_number() over(partition by dev_id order by dev_id) as rn,
avg(med1) over(order by dev_id rows between current row and 5 following) as avg_med1,
avg(med2) over(order by dev_id rows between current row and 5 following) as avg_med2
from my_table;
What I get with that query is a moving average, however, I only want the mean of the n elements for the current id. So the output should be for ID x > avg(rows 1-5), for ID x > avg(rows 6-10), for ID y > avg(rows 11-15)...
The thing im trying to replicate is something like the following:
From:
dev_id, med1, med2, med3
2, 3, 4, 1
3, 2, 1, 2
3, 1, 3, 9
3, 2, 4, 7
1, 3, 3, 2
2, 4, 3, 2
1, 5, 3, 2
3, 4, 2, 3
2, 4, 7, 2
To:
dev_id, AVG(med1), AVG(med2), AVG(med3)
2, 3.5, 3.5, 1.5
3, 1.5, 2, 5.5
3, 3, 3, 5
1, 4, 3, 2
2, 4, 5, 2

This ended up working for me, however is not what I initially planned it:
select id, reference, avg(b1), avg(b25), avg(b10), max(created_at)
from
(
select id,
#row_number := case when #reference = reference then #row_number + 1 else 0 end as row_number,
#reference := reference as reference,
b1,
b25,
b10,
created_at
from history_air
cross join (select #row_number := -1, #reference := '') as t
order by reference, created_at
) as t
group by reference, row_number div 150
order by reference, row_number div 150;

Related

DB2 SQL : selecting rows where value is different from previous one

Let's say I have a table (PERSON) like this :
I would like to select only the rows where the value of column "C" has changed from previous row.
In this case, I should get : rows 1, 4, 5, 7, 8, 9 and 15.
I can't figure out how to achieve this.
Does someone has an idea please ?
Thank you
Try this:
WITH PERSON (ROW_NUMBER, C) AS
(
VALUES
( 1, NULL::INT)
, ( 3, NULL::INT)
, ( 4, 189)
, ( 5, NULL::INT)
, ( 6, NULL::INT)
, ( 7, 212)
, ( 8, NULL::INT)
, ( 9, 235)
, (10, 235)
, (11, NULL::INT)
)
SELECT ROW_NUMBER, C
FROM
(
SELECT
P.*
, LAG (P.C) OVER (ORDER BY ROW_NUMBER) AS C_PREV
, LAG (P.ROW_NUMBER) OVER (ORDER BY ROW_NUMBER) AS ROW_NUMBER_PREV
FROM PERSON P
)
WHERE
ROW_NUMBER_PREV IS NULL
OR (C IS DISTINCT FROM C_PREV)
ROW_NUMBER
C
1
4
189
5
7
212
8
9
235
11

How to use a table in SQL WITH statement?

I am trying to use a pre-existing table in the SQL statement at the bottom of the question rather than the data that is being generated in the SQL statement. Currently, there is some data that is generated using:
WITH polys(poly_id, geom) AS (VALUES (1, 'POLYGON((1 1, 1 5, 4 5, 4 4, 2 4, 2 2, 4 2, 4 1, 1 1))'::GEOMETRY),
(2, 'POLYGON((6 6, 6 10, 8 10, 9 7, 8 6, 6 6))'::GEOMETRY)),
However, let's say I already have a table named polys with the poly_id and geom columns, exactly as what would be created above. How can I insert my pre-existing polys table into this SQL statement (i.e. what syntax would I use)?
I have tried the following to add a pre-existing polys table using:
CREATE TABLE polys_pts AS
WITH polys(poly_id, geom) AS,
with the following error:
ERROR: syntax error at or near ","
LINE 2: WITH polys(poly_id, geom) AS,
^
Full Code:
CREATE TABLE polys_pts AS
WITH polys(poly_id, geom) AS (VALUES (1, 'POLYGON((1 1, 1 5, 4 5, 4 4, 2 4, 2 2, 4 2, 4 1, 1 1))'::GEOMETRY),
(2, 'POLYGON((6 6, 6 10, 8 10, 9 7, 8 6, 6 6))'::GEOMETRY)),
pnt_clusters AS (SELECT polys.poly_id,
CASE
WHEN ST_Area(polys.geom)>9 THEN ST_ClusterKMeans(pts.geom, 8) OVER(PARTITION BY polys.poly_id)
ELSE ST_ClusterKMeans(pts.geom, 2) OVER(PARTITION BY polys.poly_id)
END AS cluster_id, pts.geom FROM polys,
LATERAL ST_Dump(ST_GeneratePoints(polys.geom, 1000, 1)) AS pts),
centroids AS (SELECT cluster_id, ST_PointOnSurface(ST_collect(geom)) AS geom FROM pnt_clusters GROUP BY poly_id, cluster_id),
neg_buffer AS (SELECT poly_id, (ST_Buffer(geom, -0.4, 'endcap=flat join=round')) geom FROM polys GROUP BY poly_id, polys.geom),
neg_buffer_pts_out AS (SELECT a.cluster_id, (a.geom) geom FROM centroids a WHERE EXISTS (SELECT 1 FROM neg_buffer b WHERE ST_Intersects(a.geom, b.geom))),
neg_buffer_pts_in AS (SELECT a.cluster_id, (a.geom) geom FROM centroids a WHERE NOT EXISTS (SELECT 1 FROM neg_buffer b WHERE ST_Intersects(a.geom, b.geom))),
snap_pts_clusters_in AS (SELECT DISTINCT ST_ClosestPoint(ST_ExteriorRing(a.geom), b.geom) AS geom FROM neg_buffer a, neg_buffer_pts_in b),
node_pts AS (SELECT ST_StartPoint(ST_ExteriorRing(geom)) geom FROM neg_buffer),
snap_pts AS (SELECT b.cluster_id, a.geom FROM snap_pts_clusters_in a JOIN centroids b ON ST_DWithin(a.geom, b.geom, 0.4))
SELECT a.cluster_id, (a.geom) geom FROM snap_pts a WHERE NOT EXISTS (SELECT 1 FROM node_pts b WHERE ST_Intersects(a.geom, b.geom))
UNION SELECT c.cluster_id, (c.geom) geom FROM neg_buffer_pts_out c ORDER BY cluster_id;
I'm not sure of understanding your question so i give you a broad answer.
To create a table from a query you must use:
CREATE TABLE foo AS
SELECT * FROM my_table;
CTEs are builded as:
WITH
tmp1 AS (
SELECT * from my_table1
), -- commna
tmp2 AS (
SELECT * from my_table2
)
SELECT * from tmp1 JOIN tmp2 ON tmp1.id = tmp2.id -- no comma
;
Note that the are , to separate different "temporary" tables defined in the CTE but the final sentence is not preceded with a ,
So to create a table from a CTE the syntax will be:
CREATE TABLE foo AS
WITH
tmp1 AS (
SELECT * from my_table1
),
tmp2 AS (
SELECT * from my_table2
)
SELECT * from tmp1 JOIN tmp2 ON tmp1.id = tmp2.id -- no comma
;
Create a table from a VALUES clause is the same as the other cases:
CREATE TABLE polys2 AS
VALUES
(1, 'POLYGON((1 1, 1 5, 4 5, 4 4, 2 4, 2 2, 4 2, 4 1, 1 1))'::GEOMETRY),
(2, 'POLYGON((6 6, 6 10, 8 10, 9 7, 8 6, 6 6))'::GEOMETRY)
;
If you already have a table called polys2 that has been created for example like is shown in the previous example, you can replace
CREATE TABLE polys_pts AS
WITH
polys(poly_id, geom) AS (
VALUES
(1, 'POLYGON((1 1, 1 5, 4 5, 4 4, 2 4, 2 2, 4 2, 4 1, 1 1))'::GEOMETRY),
(2, 'POLYGON((6 6, 6 10, 8 10, 9 7, 8 6, 6 6))'::GEOMETRY)),
pnt_clusters AS (SELECT polys.poly_id, ...
with
CREATE TABLE polys_pts AS
WITH
polys(poly_id, geom) AS (
SELECT poly_id, geom FROM polys2
),
pnt_clusters AS (SELECT polys.poly_id, ...
um, the question is not 100% clear to me - ... I am not familiar with pecularities of postgresql, but my first bet would be to try
WITH polys(...) AS (...),
pnt_clusters AS (...)
CREATE polys_pts AS (
SELECT ..
FROM polys... etc.
)
but I guess this is not allowed since WITH only goes with DML statements (data manipulation unlike data definition (DDL) statements like CREATE)
so.. my next bet would be to try using polys and pnt_clusters that you defined inside WITH clause, inline inside the SELECT statement, given that
WITH a AS (
SELECT x, y FROM z
)
SELECT *
FROM a
is the same as
SELECT *
FROM (
SELECT x, y
FROM z
) AS a
well, otherwise I would split the process into two steps - create some kind of temporary tables first for polys and pnt_clusters and then do the create...
The definition of a CTE must be a complete statement, so you have to use
WITH polys(poly_id, geom) AS (
SELECT *
FROM (VALUES
(1, 'POLYGON((1 1, 1 5, 4 5, 4 4, 2 4, 2 2, 4 2, 4 1, 1 1))'::GEOMETRY),
(2, 'POLYGON((6 6, 6 10, 8 10, 9 7, 8 6, 6 6))'::GEOMETRY)
) AS p(p, g)
)

BigQuery arrays - SELECT DISTINCT ordering guarantees?

I want to filter out the duplicates from a BigQuery array. I also need the order of the elements to be preserved. The docs mention that this can be done by combining SELECT DISTINCT with UNNEST. However, it doesn't mention any ordering behavior. I ran this query and got the desired ordering of [5, 3, 1, 4, 10, 8].
WITH an_array AS (
SELECT [5, 5, 3, 1, 4, 4, 10, 8, 5, 1] AS nums
)
SELECT
ARRAY((
SELECT DISTINCT num
FROM UNNEST(nums) num
))
FROM an_array;
I don't know if that's coincidence or if that ordering is guaranteed. I also tried adding WITH OFFSET with an ORDER BY to specify the order explicitly, but in that case I get Query error: ORDER BY clause expression references table alias offset which is not visible after SELECT DISTINCT.
You should always be explicit about ordering if you care about it:WITH an_array AS (
WITH an_array as (
SELECT [5, 5, 3, 1, 4, 4, 10, 8, 5, 1] AS nums
)
SELECT ARRAY((SELECT num
FROM UNNEST(nums) num WITH OFFSET o
GROUP BY num
ORDER BY MIN(o)
)
)
FROM an_array;

How to write sql for this case?

Suppose I have a table(a relationship) like
MyTab(ID1,ID2,IsMarked, data,....)
the sample data maybe looks like:
1, 1, 1, ...
1, 2, 0, ...
1, 3, 0, ...
2, 34, 1, ...
3, 4, 0, ...
4, 546, 0, ...
4, 8, 0, ...
Only one could be marked for each ID1. I want to get data marked as 1 for all Entities ID1. If there is no marked record, get the first one or any one of them.
For above sample data, the result should be:
1, 1, 1, ...
2, 34, 1, ...
3, 4, 0, ...
4, 546, 0, ...
Union could be a solution, but is too long and may have bad performance.
My idea is to sort the data by ID1 and IsMarked desc, the get the first 1 for each ID1, but how to write a SQL for this case?
For Only one could be marked for each ID1 the following should work:
;with cte as (
select *, rn=row_number() over (partition by ID1 order by IsMarked desc)
)
select *
from cte
where rn=1
Shot in the dark:
SELECT A.*
FROM MYTAB A
LEFT JOIN (
SELECT MAX(ID2) AS MAXID2, ID1
FROM MYTAB
WHERE ISMARKED=1
GROUP BY ID1
) B ON A.ID2=B.MAXID2 AND A.ID1=B.ID1
LEFT JOIN (
SELECT MAX(ID2) AS MAXID2, ID1
FROM MYTAB
WHERE ISMARKED=0
GROUP BY ID1
) C ON A.ID2=C.MAXID2 AND A.ID1=C.ID1
WHERE
(B.ID1 IS NOT NULL)
OR
(B.ID1 IS NULL AND C.ID1 IS NOT NULL);

How to do equivalent of "limit distinct"?

How can I limit a result set to n distinct values of a given column(s), where the actual number of rows may be higher?
Input table:
client_id, employer_id, other_value
1, 2, abc
1, 3, defg
2, 3, dkfjh
3, 1, ldkfjkj
4, 4, dlkfjk
4, 5, 342
4, 6, dkj
5, 1, dlkfj
6, 1, 34kjf
7, 7, 34kjf
8, 6, lkjkj
8, 7, 23kj
desired output, where limit distinct=5 distinct values of client_id:
1, 2, abc
1, 3, defg
2, 3, dkfjh
3, 1, ldkfjkj
4, 4, dlkfjk
4, 5, 342
4, 6, dkj
5, 1, dlkfj
Platform this is intended for is MySQL.
You can use a subselect
select * from table where client_id in
(select distinct client_id from table order by client_id limit 5)
This is for SQL Server. I can't remember, MySQL may use a LIMIT keyword instead of TOP. That may make the query more efficient if you can get rid of the inner most subquery by using the LIMIT and DISTINCT in the same subquery. (It looks like Vinko used this method and that LIMIT is correct. I'll leave this here for the second possible answer though.)
SELECT
client_id,
employer_id,
other_value
FROM
MyTable
WHERE
client_id IN
(
SELECT TOP 5
client_id
FROM
(
SELECT DISTINCT
client_id
FROM
MyTable
) SQ
ORDER BY
client_id
)
Of course, add in your own WHERE clause and ORDER BY clause in the subquery.
Another possibility (compare performance and see which works out better) is:
SELECT
client_id,
employer_id,
other_value
FROM
MyTable T1
WHERE
T1.code IN
(
SELECT
T2.code
FROM
MyTable T2
WHERE
(SELECT COUNT(*) FROM MyTable T3 WHERE T3,code < T2.code) < 5
)
-- Using Common Table Expression in Microsoft SQL Server.
-- LIMIT function does not exist in MS SQL.
WITH CTE
AS
(SELECT DISTINCT([COLUMN_NAME])
FROM [TABLE_NAME])
SELECT TOP (5) [[COLUMN_NAME]]
FROM CTE;
This works for ‍‍MS SQL if anyone is on that platform:
SET ROWCOUNT 10;
SELECT DISTINCT
column1, column2, column3,...
FROM
Table1
WHERE ...