Adding Postgres index to one table locks up another - sql

Question

I look after a single Postgres 9.3.3 (Amazon RDS instance: db.m3.2xlarge), which is the back-end of a system that logs incoming statistics and provides reports based on those data - yes, from the same DB node.
Performance is generally very good, but upon adding an extra index on table R to improve reporting performance, logging performance collapsed, as both INSERTs and UPDATEs on a different table L used by the logging process immediately began to lock - seemingly on one another, according to pg_locks, although no deadlocks were reported. Immediately, all available connections (according to pg_stat_activity) locked in the same way, DB CPU rose quickly to 100%. The logger's load-balancer took all of its nodes out of use, but as the INSERTs and UPDATEs refused to complete or to time out, all connections stayed locked.
Note that this isn't a problem during index creation, only during usage. Nor is this an issue of load: throttling logging by 90% and starting the system completely afresh again immediately locked it up. No reporting whatsoever was happening at the same time.
Dropping the R index immediately releases all L locks.
I create the index with:
CREATE INDEX idxForGroup ON R (group,article_id,month);
where the columns are:
'group' type: VARCHAR(64) defaultValue: "" nullable: false
'month' type: TIMESTAMP nullable: false
'article_id' type: BIGINT defaultValue: 0 nullable: false
There is already a composite primary key, of which the above is just a subset:
customer_resource_id (a FK), subtype (a VARCHAR), group, article_id, month
I should add that there is a relationship between R and L: a trigger updates the reporting table R based upon updates to L:
CREATE TRIGGER on_event_report AFTER INSERT OR UPDATE ON L FOR EACH ROW EXECUTE PROCEDURE resource_event_trigger();
I accept that adding any index imposes a small (microseconds?) cost/load, but there are already indexes on R, so I don't understand how a 'little' extra indexing on R could have such a huge impact as to cause lockups for L.
Update:
If I investigate the L queries that are getting locked:
EXPLAIN (analyze,buffers) update L set count=count+1 where customer_resource_id=911657 and item_type_id='type' and event_subtype='subtype' and reporting_date='2014-04-13 00:00:00' AND group='';
Update on L (cost=0.57..20.18 rows=5 width=49) (actual time=70.968..70.968 rows=0 loops=1)
Buffers: shared hit=170 read=16 dirtied=15
-> Index Scan using L_pkey on L (cost=0.57..20.18 rows=5 width=49) (actual time=0.067..0.525 rows=19 loops=1)
Index Cond: ((customer_resource_id = 911657) AND ((group)::text = ''::text) AND ((item_type_id)::text = 'type'::text) AND ((event_subtype)::text = 'subtype'::text) AND (article_id = 0))
Buffers: shared hit=24
Trigger on_L: time=11626.219 calls=19 <---
Total runtime: 11697.285 ms
So, you'd think the trigger that updates R must be the problem - and yet when I EXPLAIN the trigger queries, they all check out fine: indexes hit, no scans, etc.
Update 2:
Not sure if this is really a locking issue, or just a massive performance degradation, but here's pg_locks with the index present:
SELECT mode,COUNT(*) FROM pg_locks GROUP BY mode;
mode | granted | count
------------------+---------+-------
AccessShareLock | t | 24715
ExclusiveLock | t | 1504
ExclusiveLock | f | 138
RowExclusiveLock | t | 5901
RowShareLock | t | 185
ShareLock | f | 95
Drop the index, and within seconds:
mode | count
-----------------+-------
ExclusiveLock | 3
AccessShareLock | 31
Update 3:
Here's the source of the trigger on the logging table L that updates the reporting table R:
CREATE OR REPLACE FUNCTION resource_event_trigger()
RETURNS TRIGGER AS $$
DECLARE
cre_row R%ROWTYPE;
delta INTEGER;
BEGIN
SELECT * INTO cre_row FROM R cre WHERE cre.customer_resource_id = NEW.customer_resource_id AND cre.group = NEW.group_id AND cre.subtype = NEW.event_subtype AND cre.date = date_trunc('month', NEW.date) AND cre.article_id = NEW.article_id;
IF cre_row IS null THEN
INSERT INTO R (customer_resource_id, group, subtype, article_id, date) VALUES (NEW.customer_resource_id, NEW.group_id, NEW.event_subtype, NEW.article_id, date_trunc('month', NEW.date));
END IF;
IF TG_OP = 'INSERT' THEN
delta = NEW.event_count;
ELSE
delta = NEW.event_count - OLD.event_count;
END IF;
CASE
WHEN NEW.item_type_id = 'typeA' THEN
UPDATE R SET count_A = count_A + delta WHERE customer_resource_id = NEW.customer_resource_id AND group = NEW.group_id AND subtype = NEW.event_subtype AND article_id = NEW.article_id AND date = date_trunc('month', NEW.date);
[...]
END CASE;
RETURN NEW;
END;
$$
LANGUAGE plpgsql;
It's long-ish, but pretty straightforward. When 'EXPLAIN'ed individually, all the individual queries use primary keys / indexes, use few buffers, etc.
Update 4:
If I examine the created index, I notice:
SELECT tablename, attname, n_distinct, correlation from pg_stats where tablename='R' AND attname IN ('group','article_id','date','customer_resource_id','subtype') ORDER BY attname;
tablename | attname | n_distinct | correlation
-----------+----------------------+------------+-------------
R | article_id | 25886 | 0.756468
R | group | 165 | 0.227023
R | customer_resource_id | -0.304469 | 0.729134
R | date | 53 | 0.943593
R | subtype | 2 | 0.657429
... which looks plausible. And if I look at cardinality I get:
SELECT relname, relkind, reltuples as cardinality, relpages FROM pg_class where relkind='i' [...] order by relname;
relname | relkind | cardinality | relpages
-------------+---------+-------------+----------
R_pkey | i | 2.69955e+07 | 293035
idxForGroup | i | 2.70333e+07 | 134149
L_pkey | i | 7.14889e+07 | 771581
Both the PK and the newly added index have values that are almost the same as the row count which, again, should be fine...

Answer 1

Well while I can't say EXACTLY what your problem may be, it does obviously seem to hinge on this new index. It would certainly be easier if I were to look thoroughly through things, but I will give a couple shots in the dark.
Indexes and postgres performance can be a big subject, but fundamentally there are a few things I see could be wrong that you should check:
When changing/adding an index on a table, the query optimizer (that fires milliseconds before the query goes, and evaluates how best to execute that query) of course is looking at the table in a different way. It sees: "Hey, there is a new index here and maybe this is better than the old index I was using" and in some cases the query optimizer can be wrong.
And so instead of using a past index that was working just fine, the optimizer then starts doing full table scans or something stupid. This is in an extreme case, but could of course occur.
The other thing is that you may be dealing with a lot of "dead rows". Whenever you to an update to a table, it creates a "dead row" and inserts a new one (your updated row). The "dead row" doesn't really go anywhere and sometimes can bloat up your table and the Postgres query optimizer can go a little haywire trying to understand this.
One time I had a table that just went CRAZY like this and I could not, for the life of me, understand why it ws going SO SLOW. I then looked at the table statistics and saw hundreds of thousands of dead rows. I took a shot in the dark and just dropped and re-created the table (albeit, that takes some work on a live database) and magically everything worked completely fine after that (NOTHING was changed on the table structure or data in the table - other than that dead rows were completely released when dropping that instance of the table).
While that is a bit extreme, what I would do immediate is:
a) run vacuum on the table which does a pretty good jog at getting the "dead rows" out of the way
b) then run an analyze on the table. This re-sets the table summarized statistics, which is where the optimizer takes a lot of the data to determine how best to query the table.