Deleting millions of rows from PostgreSQL - sql

Question

Question in short
How can I delete around 30M rows from a table with 3B rows without blowing up my PostgreSQL server?
Question in detail
I am using PostgreSQL database (AWS RDS, db.t4g.medium). I have one table to keep track of daily stocks of various items of my customers, with approximately the following rows:
supplier_id: foreign key to customer table
retailer_id: foreign key to customer table
ean: varchar(13), not indexed
datetime: indexed
quantity: integer, not indexed
As of today, this table has 3 billion rows. Now I want to delete all rows for a given supplier. The naive approach would be to execute:
DELETE FROM stocks
WHERE supplier_id = 200
This will result in a non-responsive database for at least 1 hour, after which I have killed the query (since it's making my entire webserver be non-responsive).
I then split up to delete them in batches, based per day. Since I am using Python+Django, it is easy to generate these SQL queries automatically.
DELETE FROM STOCKS
WHERE supplier_id=200
AND datetime >= '2022-08-19T00:00:00+00:00'::timestamptz
AND datetime < '2022-08-20T00:00:00+00:00'::timestamptz
This had solved the issue for the smaller suppliers (1M rows), but for a supplier with 30M rows, this still blows up. After 1 hour, it didn't finish deleting for a single day.
My question is: how can I make sure that this deletion occurs without consuming all resources of the database instance? Are there more clever ways to split it up? Are there other techniques available that I don't know about? I do not care too much about how long it takes (this is not a regularly recurring operation), as long as it is stable.
Additional info
This is the query plan for the deletion of all stocks for a supplier:
EXPLAIN
DELETE FROM stocks
WHERE supplier_id = 158;
Delete on stocks (cost=0.00..80052412.00 rows=0 width=0)
-> Seq Scan on sa_inventory_stocktake (cost=0.00..80052412.00 rows=36725650 width=6)
Filter: (supplier_id = 158)
And this is the query plan for deleting the stocks of a supplier for a given day:
EXPLAIN
DELETE FROM stocks
WHERE supplier_id = 158
AND datetime >= '2022-08-19T00:00:00+00:00'::timestamptz
AND datetime < '2022-08-20T00:00:00+00:00'::timestamptz;
Delete on stocks (cost=92677221.21..92898902.60 rows=0 width=0)
-> Bitmap Heap Scan on stocks (cost=92677221.21..92898902.60 rows=56759 width=6)
Recheck Cond: ((supplier_id = 158) AND (datetime >= '2022-08-19 00:00:00+00'::timestamp with time zone) AND (datetime < '2022-08-20 00:00:00+00'::timestamp with time zone))"
-> BitmapAnd (cost=92677221.21..92677221.21 rows=56759 width=0)
-> Bitmap Index Scan on stocks_supplier_id_c50e0b94 (cost=0.00..892175.08 rows=36725650 width=0)
Index Cond: (supplier_id = 158)
-> Bitmap Index Scan on stocks_lookup (cost=0.00..91785017.50 rows=4614593 width=0)
Index Cond: ((datetime >= '2022-08-19 00:00:00+00'::timestamp with time zone) AND (datetime < '2022-08-20 00:00:00+00'::timestamp with time zone))"
There are the following indices:
BTREE on supplier_id, is_unique=False
BTREE on retailer_id, is_unique=False
BTREE on retailer_id,datetime, is_unique=False

Answer 1

The below code is just an idea. In order not to lock too many rows at a time, we do this blockwise using PostgreSQL's row location CTID. Thus we delete the first n matches, then the next n, and so on, but by using the table's CTID we avoid reading disk blocks that we have already scanned before.
I am not a PostgreSQL developer. My code doesn't work. PostgreSQL's RETURNING clause does not allow aggregation. There may be even more errors in the code. But you get the idea, and I hope you or another PostgreSQL develeoper will be able to fix all issues and get this running.
create or replace procedure delete_from_big_table(p_supplier_id int)
$$
declare
v_max_ctid tid = (0,0);
v_count int;
begin
loop
with next_rows as
(
select ctid
from big_table
where supplier_id = p_supplier_id
and ctid > v_max_ctid
order by ctid
fetch first 50000 rows only
)
delete from big_table
where big_table.ctid in (select ctid from next_rows)
returning max(ctid), count(*) into v_max_ctid, v_count;
if v_count > 0 then
commit;
else
exit;
end if;
end loop;
end;
$$ language plpgsql;
This will be slow, but who cares as long as all the other processes can run at about normal speed.