Counting number of non-trading days/days without price changes - sql

I have a table of closing prices for bonds over time, with the essential structure:
bond_id | tdate | price
---------+------------+-------
EIX1923 | 2014-01-01 | 100.12
EIX1923 | 2014-01-02 | 100.10
EIX1923 | 2014-01-05 | 100.10
EIX1923 | 2014-01-10 | 100.15
As you can see, I don't have prices for every day -- because the bond does not trade every day. I would like to count how often this occurs in a given year and, if the bond price hasn't changed between consecutive days, I take that as the same result.
That is, for a year with N trading days (excluding weekends, ignoring holidays), I would essentially want to generate a series of dates and count how many days the price is (1) unchanged from the previous day or (2) is not recored for that day and divide it over N.
I'm using PostgreSQL, so I started out with generate_series('2014-01-01'::timestamp, '2015-01-01'::timestamp, '1 day'::interval); I can SELECT from this series and do a WHERE to exclude weekends:
SELECT dd
FROM generate_series(
'2014-01-01'::timestamp,
'2015-01-01'::timestamp,
'1 day'::timestamp
) dd
WHERE EXTRACT(dow FROM dd) NOT IN (0, 6);
Now, I figure I would like to generate a "column" of bond_id to JOIN against the trade table with, but I'm not sure how. Essentially, I figured the simplest structure would be a LEFT JOIN so that I get something like:
EIX1923 | 2014-01-01 | 100.12
EIX1923 | 2014-01-02 | 100.10
EIX1923 | 2014-01-03 |
EIX1923 | 2014-01-04 |
EIX1923 | 2014-01-05 | 100.10
EIX1923 | 2014-01-06 |
EIX1923 | 2014-01-07 |
EIX1923 | 2014-01-08 |
EIX1923 | 2014-01-09 |
EIX1923 | 2014-01-10 | 100.15
Then I could just fill in the gaps with the most recently available price and count the number of ABS(∆P) == 0 in application code. But if there are solutions to do this entirely in SQL that would be nice too! I have no idea if the approach above is the right one to go with.
(I didn't bother to check if the first days of January 2014 are weekends or not, since it's just for illustration here; but they would be excluded from the results, obviously).
EDIT: Seems there might be a number of similar questions already. Hope it's not too much of a duplicate!
EDIT: So, I played a bit more with this and this solution "works" (and I feel silly for not realizing sooner) in the above sense:
SELECT
'EI653670', dd, t.price
FROM
generate_series('2014-01-01'::timestamp, '2015-01-01'::timestamp, '1 day'::interval) dd
LEFT JOIN
trade t ON dd = t.tdate AND t.id = 'EI653670'
WHERE
EXTRACT(dow FROM dd) NOT IN (0, 6) ORDER BY dd;
Is there a better way?

I think you can do this logic with lag(). The following show the general idea -- get the previous date and price and do some logic:
select bond_id,
sum(case when prev_price = price
then date - prev_date + 1
when prev_date = date + interval '1 day'
then 0
else date - prev_date
end)
from (select t.*,
lag(t.date) over (partition by t.bond_id order by t.date) as prev_date,
lag(t.price) over (partition by t.bond_id order by t.date) as prev_price
trade t
) t
group by bond_id;
One caveat is that this probably won't handle boundary conditions the way that you want.

Related

3 month rolling average with missing months

I've been reading the related questions here, and so far the solutions require that there are no missing months. Would love to get some help on what I can do if there are missing months?
For example, I'd like to calculate the 3 month rolling average of orders per item. If there is a missing month for an item, the calculation assumes that the number of orders for that item for that month is 0. If there are fewer than three months left, the rolling average isn't so important (it can be null or otherwise).
MONTH | ITEM | ORDERS | ROLLING_AVG
2021-04 | A | 5 | 3.33
2021-04 | B | 4 | 3
2021-03 | A | 3 | 1.66
2021-03 | B | 5 | null
2021-02 | A | 2 | null
2021-01 | B | 2 | null
Big thanks in advance!
Also, is there a way to "add" the missing month rows without using a cross join with a list of items? For example if I have 10 million items, the cross join takes quite a while to execute.
You can use a range window frame -- and some conditional logic:
select t.*,
(case when min(month) over (partition by item) <= month - interval '2 month'
then sum(orders) over (partition by item
order by month
range between interval '2 month' preceding and current row
) / 3.0
end) as rolling_average
from t;
Here is a db<>fiddle. The results are slightly different from what is in your question, because there is not enough info for A in 2021-03 but there is enough for B in 2021-03.

How can I aggregate values based on an arbitrary monthly cycle date range in SQL?

Given a table as such:
# SELECT * FROM payments ORDER BY payment_date DESC;
id | payment_type_id | payment_date | amount
----+-----------------+--------------+---------
4 | 1 | 2019-11-18 | 300.00
3 | 1 | 2019-11-17 | 1000.00
2 | 1 | 2019-11-16 | 250.00
1 | 1 | 2019-11-15 | 300.00
14 | 1 | 2019-10-18 | 130.00
13 | 1 | 2019-10-18 | 100.00
15 | 1 | 2019-09-18 | 1300.00
16 | 1 | 2019-09-17 | 1300.00
17 | 1 | 2019-09-01 | 400.00
18 | 1 | 2019-08-25 | 400.00
(10 rows)
How can I SUM the amount column based on an arbitrary date range, not simply a date truncation?
Taking the example of a date range beginning on the 15th of a month, and ending on the 14th of the following month, the output I would expect to see is:
payment_type_id | payment_date | amount
-----------------+--------------+---------
1 | 2019-11-15 | 1850.00
1 | 2019-10-15 | 230.00
1 | 2019-09-15 | 2600.00
1 | 2019-08-15 | 800.00
Can this be done in SQL, or is this something that's better handled in code? I would traditionally do this in code, but looking to extend my knowledge of SQL (which at this stage, isnt much!)
Click demo:db<>fiddle
You can use a combination of the CASE clause and the date_trunc() function:
SELECT
payment_type_id,
CASE
WHEN date_part('day', payment_date) < 15 THEN
date_trunc('month', payment_date) + interval '-1month 14 days'
ELSE date_trunc('month', payment_date) + interval '14 days'
END AS payment_date,
SUM(amount) AS amount
FROM
payments
GROUP BY 1,2
date_part('day', ...) gives out the current day of month
The CASE clause is for dividing the dates before the 15th of month and after.
The date_trunc('month', ...) converts all dates in a month to the first of this month
So, if date is before the 15th of the current month, it should be grouped to the 15th of the previous month (this is what +interval '-1month 14 days' calculates: +14, because the date_trunc() truncates to the 1st of month: 1 + 14 = 15). Otherwise it is group to the 15th of the current month.
After calculating these payment_days, you can use them for simple grouping.
I would simply subtract 14 days, truncate the month, and add 14 days back:
select payment_type_id,
date_trunc('month', payment_date - interval '14 day') + interval '14 day' as month_15,
sum(amount)
from payments
group by payment_type_id, month_15
order by payment_type_id, month_15;
No conditional logic is actually needed for this.
Here is a db<>fiddle.
You can use the generate_series() function and make a inner join comparing month and year, like this:
SELECT specific_date_on_month, SUM(amount)
FROM (SELECT generate_series('2015-01-15'::date, '2015-12-15'::date, '1 month'::interval) AS specific_date_on_month)
INNER JOIN payments
ON (TO_CHAR(payment_date, 'yyyymm')=TO_CHAR(specific_date_on_month, 'yyyymm'))
GROUP BY specific_date_on_month;
The generate_series(<begin>, <end>, <interval>) function generate a serie based on begin and end with an specific interval.

How to write a SQL statement to sum data using group by the same day of every two neighboring months

I have a data table like this:
datetime data
-----------------------
...
2017/8/24 6.0
2017/8/25 5.0
...
2017/9/24 6.0
2017/9/25 6.2
...
2017/10/24 8.1
2017/10/25 8.2
I want to write a SQL statement to sum the data using group by the 24th of every two neighboring months in certain range of time such as : from 2017/7/20 to 2017/10/25 as above.
How to write this SQL statement? I'm using SQL Server 2008 R2.
The expected results table is like this:
datetime_range data_sum
------------------------------------
...
2017/8/24~2017/9/24 100.9
2017/9/24~2017/10/24 120.2
...
One conceptual way to proceed here is to redefine a "month" as ending on the 24th of each normal month. Using the SQL Server month function, we will assign any date occurring after the 24th as belonging to the next month. Then we can aggregate by the year along with this shifted month to obtain the sum of data.
WITH cte AS (
SELECT
data,
YEAR(datetime) AS year,
CASE WHEN DAY(datetime) > 24
THEN MONTH(datetime) + 1 ELSE MONTH(datetime) END AS month
FROM yourTable
)
SELECT
CONVERT(varchar(4), year) + '/' + CONVERT(varchar(2), month) +
'/25~' +
CONVERT(varchar(4), year) + '/' + CONVERT(varchar(2), (month + 1)) +
'/24' AS datetime_range,
SUM(data) AS data_sum
FROM cte
GROUP BY
year, month;
Note that your suggested ranges seem to include the 24th on both ends, which does not make sense from an accounting point of view. I assume that the month includes and ends on the 24th (i.e. the 25th is the first day of the next accounting period.
Demo
I would suggest dynamically building some date range rows so that you can then join you data to those for aggregation, like this example:
+----+---------------------+---------------------+----------------+
| | period_start_dt | period_end_dt | your_data_here |
+----+---------------------+---------------------+----------------+
| 1 | 24.04.2017 00:00:00 | 24.05.2017 00:00:00 | 1 |
| 2 | 24.05.2017 00:00:00 | 24.06.2017 00:00:00 | 1 |
| 3 | 24.06.2017 00:00:00 | 24.07.2017 00:00:00 | 1 |
| 4 | 24.07.2017 00:00:00 | 24.08.2017 00:00:00 | 1 |
| 5 | 24.08.2017 00:00:00 | 24.09.2017 00:00:00 | 1 |
| 6 | 24.09.2017 00:00:00 | 24.10.2017 00:00:00 | 1 |
| 7 | 24.10.2017 00:00:00 | 24.11.2017 00:00:00 | 1 |
| 8 | 24.11.2017 00:00:00 | 24.12.2017 00:00:00 | 1 |
| 9 | 24.12.2017 00:00:00 | 24.01.2018 00:00:00 | 1 |
| 10 | 24.01.2018 00:00:00 | 24.02.2018 00:00:00 | 1 |
| 11 | 24.02.2018 00:00:00 | 24.03.2018 00:00:00 | 1 |
| 12 | 24.03.2018 00:00:00 | 24.04.2018 00:00:00 | 1 |
+----+---------------------+---------------------+----------------+
DEMO
declare #start_dt date;
set #start_dt = '20170424';
select
period_start_dt, period_end_dt, sum(1) as your_data_here
from (
select
dateadd(month,m.n,start_dt) period_start_dt
, dateadd(month,m.n+1,start_dt) period_end_dt
from (
select #start_dt start_dt ) seed
cross join (
select 0 n union all
select 1 union all
select 2 union all
select 3 union all
select 4 union all
select 5 union all
select 6 union all
select 7 union all
select 8 union all
select 9 union all
select 10 union all
select 11
) m
) r
-- LEFT JOIN YOUR DATA
-- ON yourdata.date >= r.period_start_dt and data.date < r.period_end_dt
group by
period_start_dt, period_end_dt
Please don't be tempted to use "between" when it comes to joining to your data. Follow the note above and use yourdata.date >= r.period_start_dt and data.date < r.period_end_dt otherwise you could double count information as between is inclusive of both lower and upper boundaries.
I think the simplest way is to subtract 25 days and aggregate by the month:
select year(dateadd(day, -25, datetime)) as yr,
month(dateadd(day, -25, datetime)) as mon,
sum(data)
from t
group by dateadd(day, -25, datetime);
You can format yr and mon to get the dates for the specific ranges, but this does the aggregation (and the yr/mon columns might be sufficient).
Step 0: Build a calendar table. Every database needs a calendar table eventually to simplify this sort of calculation.
In this table you may have columns such as:
Date (primary key)
Day
Month
Year
Quarter
Half-year (e.g. 1 or 2)
Day of year (1 to 366)
Day of week (numeric or text)
Is weekend (seems redundant now, but is a huge time saver later on)
Fiscal quarter/year (if your company's fiscal year doesn't start on Jan. 1)
Is Holiday
etc.
If your company starts its month on the 24th, then you can add a "Fiscal Month" column that represents that.
Step 1: Join on the calendar table
Step 2: Group by the columns in the calendar table.
Calendar tables sound weird at first, but once you realize that they are in fact tiny even if they span a couple hundred years they quickly become a major asset.
Don't try to cheap out on disk space by using computed columns. You want real columns because they are much faster and can be indexed if necessary. (Though honestly, usually just the PK index is enough for even wide calendar tables.)

Access Query - Sum of Field based on newest for each day

I have the following data:
Site # | Site Name | Product | Reading Date | Volume
1 | Cambridge | Regular | 02/21/17 08:00 | 40000
2 | Cambridge | Regular | 02/22/17 07:00 | 35000
3 | Cambridge | Regular | 02/22/17 10:00 | 30000
What I want to achieve is get the SUM of [Volume] of the last 30 days while taking the newest reading EACH day possible since its pretty inconsistent whether one day there are 1,2 or 3 readings. I have tried a couple of things but can't get it to work.
This is what I've tried:
SELECT [Site #], Product, Sum(Volume) AS SumOfVolume, DatePart("d",InventoryDate]) AS Day
FROM [Circle K New]
GROUP BY [Site #], Product, Day
HAVING (([Site #]=852446) AND (Product ="Diesel Lows"))
ORDER BY DatePart("d",[Inventory Date]) DESC;
Result:
It adds the two readings of the same day. I was/am thinking about just getting a daily average then finding the monthly average from that. But I'm unsure if the value changes affect average numbers.
Based on your description:
select sum(volume)
from data as d
where d.readingdate in (select min(d2.readingdate)
from data as d2
group by int(d2.readingdate)
) and
d.readingdate >= dateadd("d", -30, date());

Finding correlated values from second table without resorting to PL/SQL

I have the following two tables in my database:
a) A table containing values acquired at a certain date (you may think of these as, say, temperature readings):
sensor_id | acquired | value
----------+---------------------+--------
1 | 2009-04-01 10:00:00 | 20
1 | 2009-04-01 10:01:00 | 21
1 | 2009-04 01 10:02:00 | 20
1 | 2009-04 01 10:09:00 | 20
1 | 2009-04 01 10:11:00 | 25
1 | 2009-04 01 10:15:00 | 30
...
The interval between the readings may differ, but the combination of (sensor_id, acquired) is unique.
b) A second table containing time periods and a description (you may think of these as, say, periods when someone turned on the radiator):
sensor_id | start_date | end_date | description
----------+---------------------+---------------------+------------------
1 | 2009-04-01 10:00:00 | 2009-04-01 10:02:00 | some description
1 | 2009-04-01 10:10:00 | 2009-04-01 10:14:00 | something else
Again, the length of the period may differ, but there will never be overlapping time periods for any given sensor.
I want to get a result that looks like this for any sensor and any date range:
sensor id | start date | v1 | end date | v2 | description
----------+---------------------+----+---------------------+----+------------------
1 | 2009-04-01 10:00:00 | 20 | 2009-04-01 10:02:00 | 20 | some description
1 | 2009-04-01 10:10:00 | 25 | 2009-04-01 10:14:00 | 30 | some description
Or in text from: given a sensor_id and a date range of range_start and range_end,
find me all time periods which have overlap with the date range (that is, start_date < range_end and end_date > range_start) and for each of these rows, find the corresponding values from the value table for the time period's start_date and end_date (find the first row with acquired > start_date and acquired > end_date).
If it wasn't for the start_value and end_value columns, this would be a textbook trivial example of how to join two tables.
Can I somehow get the output I need in one SQL statement without resorting to writing a PL/SQL function to find these values?
Unless I have overlooked something blatantly obvious, this can't be done with simple subselects.
Database is Oracle 11g, so any Oracle-specific features are acceptable.
Edit: yes, looping is possible, but I want to know if this can be done with a single SQL select.
You can give this a try. Note the caveats at the end though.
SELECT
RNG.sensor_id,
RNG.start_date,
RDG1.value AS v1,
RNG.end_date,
RDG2.value AS v2,
RNG.description
FROM
Ranges RNG
INNER JOIN Readings RDG1 ON
RDG1.sensor_id = RNG.sensor_id AND
RDG1.acquired => RNG.start_date
LEFT OUTER JOIN Readings RDG1_NE ON
RDG1_NE.sensor_id = RDG1.sensor_id AND
RDG1_NE.acquired >= RNG.start_date AND
RDG1_NE.acquired < RDG1.acquired
INNER JOIN Readings RDG2 ON
RDG2.sensor_id = RNG.sensor_id AND
RDG2.acquired => RNG.end_date
LEFT OUTER JOIN Readings RDG1_NE ON
RDG2_NE.sensor_id = RDG2.sensor_id AND
RDG2_NE.acquired >= RNG.end_date AND
RDG2_NE.acquired < RDG2.acquired
WHERE
RDG1_NE.sensor_id IS NULL AND
RDG2_NE.sensor_id IS NULL
This uses the first reading after the start date of the range and the first reading after the end date (personally, I'd think using the last date before the start and end would make more sense or the closest value, but I don't know your application). If there is no such reading then you won't get anything at all. You can change the INNER JOINs to OUTER and put additional logic in to handle those situations based on your own business rules.
It seems pretty straight forward.
Find the sensor values for each range. Find a row - I will call acquired of this row just X - where X > start_date and not exists any other row with acquired > start_date and acquired < X. Do the same for end date.
Select only the ranges that meet the query - start_date before and end_date after the dates supplied by the query.
In SQL this would be something like that.
SELECT R1.*, SV1.aquired, SV2.aquired
FROM ranges R1
INNER JOIN sensor_values SV1 ON SV1.sensor_id = R1.sensor_id
INNER JOIN sensor_values SV2 ON SV2.sensor_id = R1.sensor_id
WHERE SV1.aquired > R1.start_date
AND NOT EXISTS (
SELECT *
FROM sensor_values SV3
WHERE SV3.aquired > R1.start_date
AND SV3.aquired < SV1.aquired)
AND SV2.aquired > R1.end_date
AND NOT EXISTS (
SELECT *
FROM sensor_values SV4
WHERE SV4.aquired > R1.end_date
AND SV4.aquired < SV2.aquired)
AND R1.start_date < #range_start
AND R1.end_date > #range_end