I have a time-series table which looks like the following:
time | a | b | c | d
--------------------+---------+----------+---------+---------
2016-05-15 00:08:22 | | | |
2016-05-15 01:50:56 | | | 26.8301 |
2016-05-15 02:41:58 | | | |
2016-05-15 03:01:37 | | | |
2016-05-15 04:45:18 | | | |
2016-05-15 05:45:32 | | | 26.9688 |
2016-05-15 06:01:48 | | | |
2016-05-15 07:47:56 | | | | 27.1269
2016-05-15 08:01:22 | | | |
2016-05-15 09:35:36 | 26.7441 | 29.8398 | | 26.9981
2016-05-15 10:08:53 | | | |
2016-05-15 11:08:30 | | | |
2016-05-15 12:14:59 | | | |
2016-05-15 13:33:36 | 27.4277 | 29.7695 | |
2016-05-15 14:36:36 | 27.4688 | 29.6836 | |
2016-05-15 15:37:36 | 27.1016 | | |
I want to return last non-null values of every column:
like this (best option):
time | column | value
--------------------+--------- +-------
2016-05-15 15:37:36 | a | 27.1016
2016-05-15 14:36:36 | b | 29.6836
2016-05-15 05:45:32 | c | 26.9688
2016-05-15 09:35:36 | d | 26.9981
like this:
column | value
-------- +-------
a | 27.1016
b | 29.6836
c | 26.9688
d | 26.9981
or at least like this:
a | b | c | d
--------+----------+---------+---------
27.1016 | 29.6836 | 26.9688 | 26.9981
Thanks!
You can unpivot and select the last row:
select distinct on (v.which) t.time, v.which, v.val
from t cross join lateral
(values (a, 'a'), (b, 'b'), (c, 'c'), (d, 'd')) v(val, which)
where v.val is not null
order by v.which, t.time desc;
I suggest another answer, but I see now that #GordonLinoff answer is better.
with src as (
select '0' as pos, 1 as a, 2 as b, null as c
union all select '1', null as a, null as b, 7 as c
union all select '2', 2 as a, null as b, 3 as c
union all select '3', null as a, null as b, null as c
union all select '4', null as a, 4 as b, null as c
),
n as (
select row_number() over() as rn, src.* from src
)
(select last_value(pos) over (order by rn desc) as timestamp, 'a' as column, last_value(a) over (order by rn desc) as value
from n
where a is not null
limit 1)
union all
(select last_value(pos) over (order by rn desc) as timestamp, 'b' as column, last_value(b) over (order by rn desc) as value
from n
where b is not null
limit 1)
union all
(select last_value(pos) over (order by rn desc) as timestamp, 'c' as column, last_value(c) over (order by rn desc) as value
from n
where c is not null
limit 1)
timestamp | column | value
:-------- | :----- | ----:
2 | a | 2
4 | b | 4
2 | c | 3
db<>fiddle here
Related
I have a dataset looks like this
| Country | id |
-------------------
| a | 5 |
| a | 1 |
| a | 2 |
| b | 1 |
| b | 5 |
| b | 4 |
| b | 7 |
| c | 5 |
| c | 1 |
| c | 2 |
and i need a query which returns 2 random values from where country in ('a', 'c'):
| Country | id |
------------------
| a | 2 | -- Two random rows from Country = 'a'
| a | 1 |
| c | 1 |
| c | 5 | --Two random rows from Country = 'c'
This should work:
select Country, id from
(select Country,
id,
row_number() over(partition by Country order by rand()) as rn
from table_name
) t
where Country in ('a', 'c') and rn <= 2
Replace rand() with random() if you're using Postgres or newid() in SQL Server.
I have the following table in Vertica:
+----------+----------+----------+
| column_1 | column_2 | column_3 |
+----------+----------+----------+
| a | 1 | 1 |
| a | 2 | 1 |
| a | 3 | 1 |
| b | 1 | 1 |
| b | 2 | 1 |
| b | 3 | 1 |
| c | 1 | 1 |
| c | 2 | 1 |
| c | 3 | 1 |
| c | 1 | 2 |
| c | 2 | 2 |
| c | 3 | 2 |
+----------+----------+----------+
The table is ordered by column_1 and column_3.
I would like to add a row number, which increases every time when column_1 or column_3 change their value. It would look something like this:
+----------+----------+----------+------------+
| column_1 | column_2 | column_3 | row_number |
+----------+----------+----------+------------+
| a | 1 | 1 | 1 |
| a | 2 | 1 | 1 |
| a | 3 | 1 | 1 |
| b | 1 | 1 | 2 |
| b | 2 | 1 | 2 |
| b | 3 | 1 | 2 |
| c | 1 | 1 | 3 |
| c | 2 | 1 | 3 |
| c | 3 | 1 | 3 |
| c | 1 | 2 | 4 |
| c | 2 | 2 | 4 |
| c | 3 | 2 | 4 |
+----------+----------+----------+------------+
I tried using partition over but I can't find the right syntax.
Vertica has the CONDITIONAL_CHANGE_EVENT() analytic functions.
It starts at 0, and increments by 1 every time the expression that makes the first argument undergoes a change.
Like so:
WITH
indata(column_1,column_2,column_3,rn) AS (
SELECT 'a',1,1,1
UNION ALL SELECT 'a',2,1,1
UNION ALL SELECT 'a',3,1,1
UNION ALL SELECT 'b',1,1,2
UNION ALL SELECT 'b',2,1,2
UNION ALL SELECT 'b',3,1,2
UNION ALL SELECT 'c',1,1,3
UNION ALL SELECT 'c',2,1,3
UNION ALL SELECT 'c',3,1,3
UNION ALL SELECT 'c',1,2,4
UNION ALL SELECT 'c',2,2,4
UNION ALL SELECT 'c',3,2,4
)
SELECT
*
, CONDITIONAL_CHANGE_EVENT(
column_1||column_3::VARCHAR
) OVER w + 1 AS rownum
FROM indata
WINDOW w AS (ORDER BY column_1,column_3,column_2)
;
-- out column_1 | column_2 | column_3 | rn | rownum
-- out ----------+----------+----------+----+--------
-- out a | 1 | 1 | 1 | 1
-- out a | 2 | 1 | 1 | 1
-- out a | 3 | 1 | 1 | 1
-- out b | 1 | 1 | 2 | 2
-- out b | 2 | 1 | 2 | 2
-- out b | 3 | 1 | 2 | 2
-- out c | 1 | 1 | 3 | 3
-- out c | 2 | 1 | 3 | 3
-- out c | 3 | 1 | 3 | 3
-- out c | 1 | 2 | 4 | 4
-- out c | 2 | 2 | 4 | 4
-- out c | 3 | 2 | 4 | 4
In the absence of an ORDER BY, SQL data sets are unordered. To establish the order in your example therefore, I've assumed the dataset can be sorted with ORDER BY column_1, column_3, column_2
If that assumption doesn't work, you MUST add additional columns that the data can be deterministically sorted by.
That gives the following query...
SELECT
yourTable.*,
DENSE_RANK() OVER (ORDER BY column_1, column_3) AS row_number
FROM
yourTable
ORDER BY
column_1, column_3, column_2
This would also work and doesn't require table sorting
Find distinct value from column_1 and column_3 and give new index for them
Merge the previous with origin table on column_1 and column_3
select t1.*, t2.row_number
from
your_table t1
join
(select column_1, column_2, row_number() over (partition by temp) as row_number from (select distinct column_1, column_2, 1 as temp from your_table) foo) t2
on
t1.column_1=t2.column_1 and t1.column_2=t2.column_2;
Suppose I have the following table:
table
| a | b | c |
|:-----|:----|:-----|
| 1 | a | NULL |
| NULL | b | NULL |
| 3 | c | NULL |
| 4 | d | 23 |
| NULL | e | 231 |
How can I count the number of NULL values by each column?
My final result would be:
| column_name | n_nulls |
|:---------------|:----------|
| a | 2 |
| b | 0 |
| c | 3 |
You can use union all:
select 'a', count(*) - count(a) as n_nulls from t
union all
select 'b', count(*) - count(b) as n_nulls from t
union all
select 'c', count(*) - count(c) as n_nulls from t;
Redshift is a column-store database, so there probably is not a more efficient method.
Is it possible to do something like:
select
avg(count(distinct user_id))
over (partition by some_date) as average_users_per_day
from user_activity
group by user_type
(notably, the partition by column, some_date, is not in the group by columns)
The idea I'm going for is something like: the average users per day by user type.
I know how to do it using subqueries (see below), but I'd like to know if there is a nice way using only over (partition by ...) and group by.
Notes:
From reading this answer, my understanding (correct me if I'm wrong) is that the following query:
select
avg(count(distinct a)) over (partition by b)
from foo
group by b
can be expanded equivalently to:
select
avg(count_distinct_a)
from (
select
b,
count(distinct a) as count_distinct_a
from foo
group by b
)
group by b
And from that, I can tweak it a bit to achieve what I want:
select
avg(count_distinct_user_id) as average_users_per_day
from (
select
user_type,
count(distinct user_id) as count_distinct_user_id
from user_activity
group by user_type, some_date
)
group by user_type
(notably, the inner group by user_type, some_date differs from the outer group by user_type)
I'd like to be able to tell the partition by-group by interaction to use a "sub-group-by" for the windowing part. Please let me know if my understanding of partition by/group by is completely off.
EDIT: Some sample data and desired output.
Source table:
+---------+-----------+-----------+
| user_id | user_type | some_date |
+---------+-----------+-----------+
| 1 | a | 1 |
| 1 | a | 2 |
| 2 | a | 1 |
| 3 | a | 2 |
| 3 | a | 2 |
| 4 | b | 2 |
| 5 | b | 1 |
| 5 | b | 3 |
| 5 | b | 3 |
| 6 | c | 1 |
| 7 | c | 1 |
| 8 | c | 4 |
| 9 | c | 2 |
| 9 | c | 3 |
| 9 | c | 4 |
+---------+-----------+-----------+
Sample intermediate table (for reasoning with):
+-----------+-----------+---------------------+
| user_type | some_date | distinct_user_count |
+-----------+-----------+---------------------+
| a | 1 | 2 |
| a | 2 | 2 |
| b | 1 | 1 |
| b | 2 | 1 |
| b | 3 | 1 |
| c | 1 | 2 |
| c | 2 | 1 |
| c | 3 | 1 |
| c | 4 | 2 |
+-----------+-----------+---------------------+
SQL is: select user_type, some_date, count(distinct user_id) from user_activity group by user_type, some_date.
Desired result:
+-----------+---------------------+
| user_type | average_daily_users |
+-----------+---------------------+
| a | 2 |
| b | 1 |
| c | 1.5 |
+-----------+---------------------+
I've got a table of temperature samples over time from several sources and I want to find the minimum, maximum, and average temperatures across all sources at set time intervals. At first glance this is easily done like so:
SELECT MIN(temp), MAX(temp), AVG(temp) FROM samples GROUP BY time;
However, things become much more complicated (to the point of where I'm stumped!) if sources drop in and out and rather than ignoring the missing sources during the intervals in question I want to use the sources' last know temperatures for the missing samples. Using datetimes and constructing intervals (say every minute) across samples unevenly distributed over time further complicates things.
I think it should be possible to create the results I want by doing a self-join on the samples table where the time from the first table is greater than or equal to the time of the second table and then calculating aggregate values for rows grouped by source. However, I'm stumped about how to actually do this.
Here's my test table:
+------+------+------+
| time | source | temp |
+------+------+------+
| 1 | a | 20 |
| 1 | b | 18 |
| 1 | c | 23 |
| 2 | b | 21 |
| 2 | c | 20 |
| 2 | a | 18 |
| 3 | a | 16 |
| 3 | c | 13 |
| 4 | c | 15 |
| 4 | a | 4 |
| 4 | b | 31 |
| 5 | b | 10 |
| 5 | c | 16 |
| 5 | a | 22 |
| 6 | a | 18 |
| 6 | b | 17 |
| 7 | a | 20 |
| 7 | b | 19 |
+------+------+------+
INSERT INTO samples (time, source, temp) VALUES (1, 'a', 20), (1, 'b', 18), (1, 'c', 23), (2, 'b', 21), (2, 'c', 20), (2, 'a', 18), (3, 'a', 16), (3, 'c', 13), (4, 'c', 15), (4, 'a', 4), (4, 'b', 31), (5, 'b', 10), (5, 'c', 16), (5, 'a', 22), (6, 'a', 18), (6, 'b', 17), (7, 'a', 20), (7, 'b', 19);
To do my min, max and avg calculations, I want an intermediate table that looks like this:
+------+------+------+
| time | source | temp |
+------+------+------+
| 1 | a | 20 |
| 1 | b | 18 |
| 1 | c | 23 |
| 2 | b | 21 |
| 2 | c | 20 |
| 2 | a | 18 |
| 3 | a | 16 |
| 3 | b | 21 |
| 3 | c | 13 |
| 4 | c | 15 |
| 4 | a | 4 |
| 4 | b | 31 |
| 5 | b | 10 |
| 5 | c | 16 |
| 5 | a | 22 |
| 6 | a | 18 |
| 6 | b | 17 |
| 6 | c | 16 |
| 7 | a | 20 |
| 7 | b | 19 |
| 7 | c | 16 |
+------+------+------+
The following query is getting me close to what I want but it takes the temperature value of the source's first result, rather than the most recent one at the given time interval:
SELECT s.dt as sdt, s.mac, ss.temp, MAX(ss.dt) as maxdt FROM (SELECT DISTINCT dt FROM samples) AS s CROSS JOIN samples AS ss WHERE s.dt >= ss.dt GROUP BY sdt, mac HAVING maxdt <= s.dt ORDER BY sdt ASC, maxdt ASC;
+------+------+------+-------+
| sdt | mac | temp | maxdt |
+------+------+------+-------+
| 1 | a | 20 | 1 |
| 1 | c | 23 | 1 |
| 1 | b | 18 | 1 |
| 2 | a | 20 | 2 |
| 2 | c | 23 | 2 |
| 2 | b | 18 | 2 |
| 3 | b | 18 | 2 |
| 3 | a | 20 | 3 |
| 3 | c | 23 | 3 |
| 4 | a | 20 | 4 |
| 4 | c | 23 | 4 |
| 4 | b | 18 | 4 |
| 5 | a | 20 | 5 |
| 5 | c | 23 | 5 |
| 5 | b | 18 | 5 |
| 6 | c | 23 | 5 |
| 6 | a | 20 | 6 |
| 6 | b | 18 | 6 |
| 7 | c | 23 | 5 |
| 7 | b | 18 | 7 |
| 7 | a | 20 | 7 |
+------+------+------+-------+
Update: chadhoc (great name, by the way!) gives a nice solution that unfortunately does not work in MySQL, since it does not support the FULL JOIN he uses. Luckily, I believe a simple UNION is an effective replacement:
-- Unify the original samples with the missing values that we've calculated
(
SELECT time, source, temp
FROM samples
)
UNION
( -- Pull all the time/source combinations that we are missing from the sample set, along with the temp
-- from the last sampled interval for the same time/source combination if we do not have one
SELECT a.time, a.source, (SELECT t2.temp FROM samples AS t2 WHERE t2.time < a.time AND t2.source = a.source ORDER BY t2.time DESC LIMIT 1) AS temp
FROM
( -- All values we want to get should be a cross of time/temp
SELECT t1.time, s1.source
FROM
(SELECT DISTINCT time FROM samples) AS t1
CROSS JOIN
(SELECT DISTINCT source FROM samples) AS s1
) AS a
LEFT JOIN samples s
ON a.time = s.time
AND a.source = s.source
WHERE s.source IS NULL
)
ORDER BY time, source;
Update 2: MySQL gives the following EXPLAIN output for chadhoc's code:
+----+--------------------+------------+------+---------------+------+---------+------+------+-----------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+--------------------+------------+------+---------------+------+---------+------+------+-----------------------------+
| 1 | PRIMARY | temp | ALL | NULL | NULL | NULL | NULL | 18 | |
| 2 | UNION | <derived4> | ALL | NULL | NULL | NULL | NULL | 21 | |
| 2 | UNION | s | ALL | NULL | NULL | NULL | NULL | 18 | Using where |
| 4 | DERIVED | <derived6> | ALL | NULL | NULL | NULL | NULL | 3 | |
| 4 | DERIVED | <derived5> | ALL | NULL | NULL | NULL | NULL | 7 | |
| 6 | DERIVED | temp | ALL | NULL | NULL | NULL | NULL | 18 | Using temporary |
| 5 | DERIVED | temp | ALL | NULL | NULL | NULL | NULL | 18 | Using temporary |
| 3 | DEPENDENT SUBQUERY | t2 | ALL | NULL | NULL | NULL | NULL | 18 | Using where; Using filesort |
| NULL | UNION RESULT | <union1,2> | ALL | NULL | NULL | NULL | NULL | NULL | Using filesort |
+----+--------------------+------------+------+---------------+------+---------+------+------+-----------------------------+
I was able to get Charles' code working like so:
SELECT T.time, S.source,
COALESCE(
D.temp,
(
SELECT temp FROM samples
WHERE source = S.source AND time = (
SELECT MAX(time)
FROM samples
WHERE
source = S.source
AND time < T.time
)
)
) AS temp
FROM (SELECT DISTINCT time FROM samples) AS T
CROSS JOIN (SELECT DISTINCT source FROM samples) AS S
LEFT JOIN samples AS D
ON D.source = S.source AND D.time = T.time
Its explanation is:
+----+--------------------+------------+------+---------------+------+---------+------+------+-----------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+--------------------+------------+------+---------------+------+---------+------+------+-----------------+
| 1 | PRIMARY | <derived5> | ALL | NULL | NULL | NULL | NULL | 3 | |
| 1 | PRIMARY | <derived4> | ALL | NULL | NULL | NULL | NULL | 7 | |
| 1 | PRIMARY | D | ALL | NULL | NULL | NULL | NULL | 18 | |
| 5 | DERIVED | temp | ALL | NULL | NULL | NULL | NULL | 18 | Using temporary |
| 4 | DERIVED | temp | ALL | NULL | NULL | NULL | NULL | 18 | Using temporary |
| 2 | DEPENDENT SUBQUERY | temp | ALL | NULL | NULL | NULL | NULL | 18 | Using where |
| 3 | DEPENDENT SUBQUERY | temp | ALL | NULL | NULL | NULL | NULL | 18 | Using where |
+----+--------------------+------------+------+---------------+------+---------+------+------+-----------------+
I think you'll get better performance making use of the ranking/windowing functions in mySql, but unfortunately I do not know those as well as the TSQL implementation. Here is an ANSI compliant solution that will work though:
-- Full join across the sample set and anything missing from the sample set, pulling the missing temp first if we do not have one
select coalesce(c1.[time], c2.[time]) as dt, coalesce(c1.source, c2.source) as source, coalesce(c2.temp, c1.temp) as temp
from samples c1
full join ( -- Pull all the time/source combinations that we are missing from the sample set, along with the temp
-- from the last sampled interval for the same time/source combination if we do not have one
select a.time, a.source,
(select top 1 t2.temp from samples t2 where t2.time < a.time and t2.source = a.source order by t2.time desc) as temp
from
( -- All values we want to get should be a cross of time/samples
select t1.[time], s1.source
from
(select distinct [time] from samples) as t1
cross join
(select distinct source from samples) as s1
) a
left join samples s
on a.[time] = s.time
and a.source = s.source
where s.source is null
) c2
on c1.time = c2.time
and c1.source = c2.source
order by dt, source
I know this looks complicated, but it's formatted to explain itself...
It should work... Hope you only have three sources... If you have an arbitrary number of sources than this won't work... In that case see the second query...
EDIT: Removed first attempt
EDIT: If you don't know the sources ahead of time, you'll have to do something where you create an intermediate result set that "Fills in" the missing values..
something like this:
2nd EDIT: Removed need for Coalesce by moving logic to retrieve most recent temp reading for each source from Select clause into the Join condition.
Select T.Time, Max(Temp) MaxTemp,
Min(Temp) MinTemp, Avg(Temp) AvgTemp
From
(Select T.TIme, S.Source, D.Temp
From (Select Distinct Time From Samples) T
Cross Join
(Select Distinct Source From Samples) S
Left Join Samples D
On D.Source = S.Source
And D.Time =
(Select Max(Time)
From Samples
Where Source = S.Source
And Time <= T.Time)) Z
Group By T.Time