I have prepared a simple SQL Fiddle demonstrating my problem -
In PostgreSQL 10.3 I store user information, two-player games and the moves in the following 3 tables:
CREATE TABLE players (
uid SERIAL PRIMARY KEY,
name text NOT NULL
);
CREATE TABLE games (
gid SERIAL PRIMARY KEY,
player1 integer NOT NULL REFERENCES players ON DELETE CASCADE,
player2 integer NOT NULL REFERENCES players ON DELETE CASCADE
);
CREATE TABLE moves (
mid BIGSERIAL PRIMARY KEY,
uid integer NOT NULL REFERENCES players ON DELETE CASCADE,
gid integer NOT NULL REFERENCES games ON DELETE CASCADE,
played timestamptz NOT NULL
);
Let's assume that 2 players, Alice and Bob have played 3 games with each other:
INSERT INTO players (name) VALUES ('Alice'), ('Bob');
INSERT INTO games (player1, player2) VALUES (1, 2);
INSERT INTO games (player1, player2) VALUES (1, 2);
INSERT INTO games (player1, player2) VALUES (1, 2);
And let's assume that the 1st game was played quickly, with moves being played every minute.
But then they chilled :-) and played 2 slow games, with moves every 10 minutes:
INSERT INTO moves (uid, gid, played) VALUES
(1, 1, now() + interval '1 min'),
(2, 1, now() + interval '2 min'),
(1, 1, now() + interval '3 min'),
(2, 1, now() + interval '4 min'),
(1, 1, now() + interval '5 min'),
(2, 1, now() + interval '6 min'),
(1, 2, now() + interval '10 min'),
(2, 2, now() + interval '20 min'),
(1, 2, now() + interval '30 min'),
(2, 2, now() + interval '40 min'),
(1, 2, now() + interval '50 min'),
(2, 2, now() + interval '60 min'),
(1, 3, now() + interval '110 min'),
(2, 3, now() + interval '120 min'),
(1, 3, now() + interval '130 min'),
(2, 3, now() + interval '140 min'),
(1, 3, now() + interval '150 min'),
(2, 3, now() + interval '160 min');
At a web page with gaming statistics I would like to display average time passing between moves for each player.
So I suppose I have to use the LAG window function of PostgreSQL.
Since several games can be played simultaneously, I am trying to PARTITION BY gid (i.e. by the "game id").
Unfortunately, I get a syntax error window function calls cannot be nested with my SQL query:
SELECT AVG(played - LAG(played) OVER (PARTITION BY gid order by played))
OVER (PARTITION BY gid order by played)
FROM moves
-- trying to calculate average thinking time for player Alice
WHERE uid = 1;
UPDATE:
Since the number of games in my database is large and grows day by day, I have tried (here the new SQL Fiddle) adding a condition to the inner select query:
SELECT AVG(played - prev_played)
FROM (SELECT m.*,
LAG(m.played) OVER (PARTITION BY m.gid ORDER BY played) AS prev_played
FROM moves m
JOIN games g ON (m.uid in (g.player1, g.player2))
WHERE m.played > now() - interval '1 month'
) m
WHERE uid = 1;
However for some reason this changes the returned value quite radically to 1 min 45 sec.
And I wonder, why does the inner SELECT query suddenly return much more rows, is maybe some condition missing in my JOIN?
UPDATE 2:
Oh ok, I get why the average value decreases: through multiple rows with same timestamps (i.e. played - prev_played = 0), but how to fix the JOIN?
UPDATE 3:
Nevermind, I was missing the m.gid = g.gid AND condition in my SQL JOIN, now it works:
SELECT AVG(played - prev_played)
FROM (SELECT m.*,
LAG(m.played) OVER (PARTITION BY m.gid ORDER BY played) AS prev_played
FROM moves m
JOIN games g ON (m.gid = g.gid AND m.uid in (g.player1, g.player2))
WHERE m.played > now() - interval '1 month'
) m
WHERE uid = 1;
You need subqueries to nest the window functions. I think this does what you want:
select avg(played - prev_played)
from (select m.*,
lag(m.played) over (partition by gid order by played) as prev_played
from moves m
) m
where uid = 1;
Note: The where needs to go in the outer query, so it doesn't affect the lag().
Probably #gordon answer is good enough. But that isn't the result you ask in your comment. Only works because the data have same number of rows for each game so average of games is the same as complete average. But if you want average of the games you need one additional level.
With cte as (
SELECT gid, AVG(played - prev_played) as play_avg
FROM (select m.*,
lag(m.played) over (partition by gid order by played) as prev_played
from moves m
) m
WHERE uid = 1
GROUP BY gid
)
SELECT AVG(play_avg)
FROM cte
;
Related
How to get a continuous date interval from rows fulfilling specific condition?
I have a table of employees states with 2 types of user_position.
The interval is continuous if the next higher date_position per user_id has the same user_id, the next day value and user_position didn't change. The user cannot have different user positions in one day.
Have a feeling it requires several cases, window functions and tsrange, but can't quite get the right result.
I would be really grateful if you could help me.
Fiddle:
http://sqlfiddle.com/#!17/ba641/1/0
The result should look like this:
user_id
user_position
position_start
position_end
1
1
01.01.2019
02.01.2019
1
2
03.01.2019
04.01.2019
1
1
05.01.2019
06.01.2019
2
1
01.01.2019
03.01.2019
2
2
04.01.2019
05.01.2019
2
2
08.01.2019
08.01.2019
2
2
10.01.2019
10.01.2019
Create/insert query for the source data:
CREATE TABLE IF NOT EXISTS users_position
( id integer GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
user_id integer,
user_position integer,
date_position date);
INSERT INTO users_position (user_id,
user_position,
date_position)
VALUES
(1, 1, '2019-01-01'),
(1, 1, '2019-01-02'),
(1, 2, '2019-01-03'),
(1, 2, '2019-01-04'),
(1, 1, '2019-01-05'),
(1, 1, '2019-01-06'),
(2, 1, '2019-01-01'),
(2, 1, '2019-01-02'),
(2, 1, '2019-01-03'),
(2, 2, '2019-01-04'),
(2, 2, '2019-01-05'),
(2, 2, '2019-01-08'),
(2, 2, '2019-01-10');
SELECT user_id, user_position
, min(date_position) AS position_start
, max(date_position) AS position_end
FROM (
SELECT user_id, user_position,date_position
, count(*) FILTER (WHERE (date_position = last_date + 1
AND user_position = last_pos) IS NOT TRUE)
OVER (PARTITION BY user_id ORDER BY date_position) AS interval
FROM (
SELECT user_id, user_position, date_position
, lag(date_position) OVER w AS last_date
, lag(user_position) OVER w AS last_pos
FROM users_position
WINDOW w AS (PARTITION BY user_id ORDER BY date_position)
) sub1
) sub2
GROUP BY user_id, user_position, interval
ORDER BY user_id, interval;
db<>fiddle here
Basically, this forms intervals by counting the number of disruptions in continuity. Whenever the "next" row per user_id is not what's expected, a new interval starts.
The WINDOW clause allows to specify a window frame once and use it repeatedly; no effect on performance.
last_date + 1 works while last_date is type date. See:
Is there a way to do date arithmetic on values of type DATE without result being of type TIMESTAMP?
Related:
Get start and end date time based on based on sequence of rows
Select longest continuous sequence
About the aggregate FILTER:
Aggregate columns with additional (distinct) filters
So for the following schema:
CREATE TABLE activity (
id integer NOT NULL,
start_date date NOT NULL
);
CREATE TABLE account (
id integer NOT NULL,
name varchar NOT NULL
);
CREATE TABLE contact (
id integer NOT NULL,
account_id integer NOT NULL,
name varchar NOT NULL
);
CREATE TABLE activity_contact (
id integer NOT NULL,
contact_id integer NOT NULL,
activity_id integer NOT NULL
);
insert into activity(id, start_date)
values
(1, '2021-11-03'),
(2, '2021-10-03'),
(3, '2021-11-02');
insert into account(id, name)
values
(1, 'Test Account');
insert into contact(id, account_id, name)
values
(1, 1, 'John'),
(2, 1, 'Kevin');
insert into activity_contact(id, contact_id, activity_id)
values
(1, 1, 1),
(2, 2, 1),
(3, 2, 2),
(4, 1, 3);
You can see that there are 3 activities and each contact has two. What i am searching for is the number of activities per account in the previous two months. So I have the following query
SELECT contact.account_id AS accountid,
count(*) FILTER (WHERE date_trunc('month'::text, activity.start_date) = date_trunc('month'::text, CURRENT_DATE - '1 mon'::interval)) AS last_month,
count(*) FILTER (WHERE date_trunc('month'::text, activity.start_date) = date_trunc('month'::text, CURRENT_DATE - '2 mons'::interval)) AS prev_month
FROM activity
JOIN activity_contact ON activity_contact.activity_id = activity.id
JOIN contact ON contact.id = activity_contact.contact_id
JOIN account ON contact.account_id = account.id
GROUP BY contact.account_id;
This returns:
accountid last_month prev_month
1 3 1
However this is incorrect. There are only 3 activities, its just that each contact sees activity 1. so it is counting that activity twice. Is there a way for me to only count each activity id one time so there is no duplication?
count(DISTINCT activity_id) to fold duplicates in the count, like Edouard suggested.
But there is more:
SELECT con.account_id AS accountid
, count(DISTINCT aco.activity_id) FILTER (WHERE act.start_date >= date_trunc('month', LOCALTIMESTAMP - interval '1 mon')
AND act.start_date < date_trunc('month', LOCALTIMESTAMP)) AS last_month
, count(DISTINCT aco.activity_id) FILTER (WHERE act.start_date >= date_trunc('month', LOCALTIMESTAMP - interval '2 mon')
AND act.start_date < date_trunc('month', LOCALTIMESTAMP - interval '1 mon')) AS prev_month
FROM activity act
JOIN activity_contact aco ON aco.activity_id = act.id
AND act.start_date >= date_trunc('month', LOCALTIMESTAMP - interval '2 mon')
AND act.start_date < date_trunc('month', LOCALTIMESTAMP)
RIGHT JOIN contact con ON con.id = aco.contact_id
-- JOIN account acc ON con.account_id = acc.id -- noise
GROUP BY 1;
db<>fiddle here
Most importantly, add an outer WHERE clause to the query to filter irrelevant rows early. This can make a big difference for a small selection from a big table.
We have to move that predicate to the JOIN clause, lest we'd exclude accounts with no activity. (LEFT JOIN and RIGHT JOIN can both be used, mirroring each other.)
See:
Postgres Left Join with where condition
Explain JOIN vs. LEFT JOIN and WHERE condition performance suggestion in more detail
Make that filter "sargable", so it can use an index on (start_date) (unlike your original formulation). Again, big impact for a small selection from a big table.
Use the same expressions for your aggregate filter clauses. Lesser effect, but take it.
Unlike other aggregate functions, count() returns 0 (not NULL) for "no rows", so we don't have to do anything extra.
Assuming referential integrity (enforced with a FK constraint), the join to table account is just expensive noise. Drop it.
CURRENT_DATE is not wrong. But since your expressions yield timestamp anyway, it's bit more efficient to use LOCALTIMESTAMP to begin with.
Compare with your original to see that this is quite a bit faster.
And I assume you are aware that this query introduces a dependency on the TimeZone setting of the executing session. The current date depends on where in the world we ask. See:
Ignoring time zones altogether in Rails and PostgreSQL
If you are not bound to this particular output format, a pivoted form is simpler, now that we filter rows early:
SELECT con.account_id AS accountid
, date_trunc('month', act.start_date) AS mon
, count(DISTINCT aco.activity_id) AS dist_count
FROM activity act
JOIN activity_contact aco ON aco.activity_id = act.id
AND act.start_date >= date_trunc('month', LOCALTIMESTAMP - interval '2 mon')
AND act.start_date < date_trunc('month', LOCALTIMESTAMP)
RIGHT JOIN contact con ON con.id = aco.contact_id
GROUP BY 1, 2
ORDER BY 1, 2 DESC;
Again, we can include accounts without activity. But months without activity do not show up ...
create table active_users(
user_id numeric,
movie_streamed date
)
insert into active_users values (1,'2020-01-2'::date);
insert into active_users values (1,'2020-01-9'::date);
insert into active_users values (1,'2020-01-16'::date);
insert into active_users values (1,'2020-01-23'::date);
insert into active_users values (1,'2020-01-30'::date);
insert into active_users values (2,'2020-01-14'::date);
insert into active_users values (2,'2020-01-16'::date);
Hi all,
I am looking for a query which will help me to get the users who watched at least a movie every week in this month(being the test data). Given the data where every record has the user_id and when that particular person has watched the movie given the date. I want a generic answer. Not like every month has 4 weeks. Because there could be some scenarios where there are 5 weeks in some cases too.
You can use generate_series(1,5) by counting from 1 upto 5, since there should 5 different weeks might exist even uncompleted as you already mentioned.
The trick is to compare the distinct count of the beginning dates for each week within the current month :
SELECT u.user_id
FROM active_users u
JOIN generate_series( 1, 5 ) g
ON date_trunc('week', movie_streamed)
= date_trunc('week', current_date) + interval '7' day * (g-1)
GROUP BY u.user_id
HAVING COUNT(DISTINCT date_trunc('week', movie_streamed)) =
(
SELECT COUNT(*)
FROM generate_series( 1, 5 ) g
WHERE to_char(current_date,'yyyymm')
= to_char(date_trunc('week', current_date)
+ interval '7' day * (g-1),'yyyymm')
);
Demo
In my PostgreSQL database I have the following schema:
CREATE TABLE programs (
id integer,
description text
);
CREATE TABLE public.messages (
id integer,
program_id integer,
text text,
created_at timestamp with time zone
);
INSERT INTO programs VALUES(1, 'Test program');
INSERT INTO messages VALUES(1,1, 'Test message 1', now() - interval '7 days');
INSERT INTO messages VALUES(2,1, 'Test message 2', now() - interval '4 days');
INSERT INTO messages VALUES(3,1, 'Test message 3', now() - interval '1 days');
I want to calculate gaps between created_at in messages table. It should work this way:
Calculate gap between created_at of first and second message.
Calculate gap between created_at of second and third message.
Calculate average gap based on those values.
Is there any way of doing such a thing in PostgreSQL?
https://www.db-fiddle.com/f/gvxijmp8u6wr6mYcSoAeVV/0
Using LAG and windowed AVG to get both difference and average gap:
WITH cte AS (
SELECT *,
created_at-LAG(created_at) OVER(PARTITION BY program_id ORDER BY created_at) gap
FROM messages
)
SELECT *, AVG(gap) OVER(PARTITION BY program_id) AS avg_gap
FROM cte;
db<>fiddle demo
If you want the average time between messages, there is no need to get the successive differences. Simply do look at the oldest and newest messages:
select program_id,
(max(created_at) - min(created_at)) / nullif(count(*) - 1, 0)
from messages
group by program_id;
I have the following event table in Postgres 9.3:
CREATE TABLE event (
event_id integer PRIMARY KEY,
user_id integer,
event_type varchar,
event_time timestamptz
);
My goal is to retrieve all user_id's with a gap of at least 30 days between any of their events (or between their last event and the current time). An additional complication is that I only want the users who have one of these gaps occur at a later time than them performing a certain event_type 'convert'. How can this be done easily?
Some example data in the event table might look like:
INSERT INTO event (event_id, user_id, event_type, event_time)
VALUES
(10, 1, 'signIn', '2015-05-05 00:11'),
(11, 1, 'browse', '2015-05-05 00:12'), -- no 'convert' event
(20, 2, 'signIn', '2015-06-07 02:35'),
(21, 2, 'browse', '2015-06-07 02:35'),
(22, 2, 'convert', '2015-06-07 02:36'), -- only 'convert' event
(23, 2, 'signIn', '2015-08-10 11:00'), -- gap of >= 30 days
(24, 2, 'signIn', '2015-08-11 11:00'),
(30, 3, 'convert', '2015-08-07 02:36'), -- starting with 1st 'convert' event
(31, 3, 'signIn', '2015-08-07 02:36'),
(32, 3, 'convert', '2015-08-08 02:36'),
(33, 3, 'signIn', '2015-08-12 11:00'), -- all gaps below 30 days
(33, 3, 'browse', '2015-08-12 11:00'), -- gap until today (2015-08-20) too small
(40, 4, 'convert', '2015-05-07 02:36'),
(41, 4, 'signIn', '2015-05-12 11:00'); -- gap until today (2015-08-20) >= 30 days
Expected result:
user_id
--------
2
4
One way to do it:
SELECT user_id
FROM (
SELECT user_id
, lead(e.event_time, 1, now()) OVER (PARTITION BY e.user_id ORDER BY e.event_time)
- event_time AS gap
FROM ( -- only users with 'convert' event
SELECT user_id, min(event_time) AS first_time
FROM event
WHERE event_type = 'convert'
GROUP BY 1
) e1
JOIN event e USING (user_id)
WHERE e.event_time >= e1.first_time
) sub
WHERE gap >= interval '30 days'
GROUP BY 1;
The window function lead() allows to include a default value if there is no "next row", which is convenient to cover your additional requirement "or between their last event and the current time".
Indexes
You should at least have an index on (user_id, event_time) if your table is big:
CREATE INDEX event_user_time_idx ON event(user_id, event_time);
If you do that often and the event_type 'convert' is rare, add another partial index:
CREATE INDEX event_user_time_convert_idx ON event(user_id, event_time)
WHERE event_type = 'convert';
For many events per user
And only if gaps of 30 days are common (not a rare case).
Indexes become even more important.
Try this recursive CTE for better performance:
WITH RECURSIVE cte AS (
( -- parentheses required
SELECT DISTINCT ON (user_id)
user_id, event_time, interval '0 days' AS gap
FROM event
WHERE event_type = 'convert'
ORDER BY user_id, event_time
)
UNION ALL
SELECT c.user_id, e.event_time, COALESCE(e.event_time, now()) - c.event_time
FROM cte c
LEFT JOIN LATERAL (
SELECT e.event_time
FROM event e
WHERE e.user_id = c.user_id
AND e.event_time > c.event_time
ORDER BY e.event_time
LIMIT 1 -- the next later event
) e ON true -- add 1 row after last to consider gap till "now"
WHERE c.event_time IS NOT NULL
AND c.gap < interval '30 days'
)
SELECT * FROM cte
WHERE gap >= interval '30 days';
It has considerably more overhead, but can stop - per user - at the first gap that's big enough. If that should be the gap between the last event now, then event_time in the result is NULL.
New SQL Fiddle with more revealing test data demonstrating both queries.
Detailed explanation in these related answers:
Optimize GROUP BY query to retrieve latest record per user
Select first row in each GROUP BY group?
SQL Fiddle
This is another way, probably not as neat as #Erwin but have all the step separated so is easy to adapt.
include_today: add a dummy event to indicate current date.
event_convert: calculate the first time the event convert appear for each user_id (in this case only user_id = 2222)
event_row: asign an unique consecutive id to each event. starting from 1 for each user_id
last part join all together and using rnum = rnum + 1 so could calculate date difference.
also the result show both event involve in the 30 days range so you can see if that is the result you want.
.
WITH include_today as (
(SELECT 'xxxx' event_id, user_id, 'today' event_type, current_date as event_time
FROM users)
UNION
(SELECT *
FROM event)
),
event_convert as (
SELECT user_id, MIN(event_time) min_time
FROM event
WHERE event_type = 'convert'
GROUP BY user_id
),
event_row as (
SELECT *, row_number() OVER (PARTITION BY user_id ORDER BY event_time desc) as rnum
FROM
include_today
)
SELECT
A.user_id,
A.event_id eventA,
A.event_type typeA,
A.event_time timeA,
B.event_id eventB,
B.event_type typeB,
B.event_time timeB,
(B.event_time - A.event_time) days
FROM
event_convert e
Inner Join event_row A
ON e.user_id = A.user_id and e.min_time <= a. event_time
Inner Join event_row B
ON A.rnum = B.rnum + 1
AND A.user_id = B.user_id
WHERE
(B.event_time - A.event_time) > interval '30 days'
ORDER BY 1,4