Alternative to Postgres SERIAL field to solve incrementing values when ON CONFLICT causes update - sql

Question

I've recently been caught out by being unaware of the issue where SERIAL fields increment whether data is inserted or not.
Most of the answers I've read on this matter discuss preventing holes from appearing in the column, which I'm fairly certain in most cases isn't what the question posed is concerned with, and it certainly wasn't in my case.
My situation was that a specific user of my software was using a feature in a way that caused millions of upserts to be performed on a single record. That record was used as status information, and in my naivety I was blissfully unaware of the impending failure when the INTEGER id fields nextval() reached its limit, that being the following error:
ERROR: integer out of range
SQL state: 22003
So my question is and was, how can I prevent id fields from incrementing the next sequence value in the case of a conflict rollback.
I look foward to others adding their knowledge to my solution.

Answer 1

My immediate solution to this issue which alleviated the out of range situation was to alter the column to BIGINT, as follows:
ALTER TABLE MyTable ALTER COLUMN idMyTable TYPE BIGINT;
The number of records in my case was extremely small (<1000) so this was a trivial alteration to perform.
Once that was out of the way, it was time to look for a solution to the underlying issue. My solution is unlikely to be as performant as using a SERIAL field, so keep that in mind based on your use case if you are going to implement something similar to what I have done - there's always a trade off somewhere.
Consider the following table and resulting data insert/query:
CREATE TABLE TestTable ( id SERIAL PRIMARY KEY NOT NULL, Key TEXT UNIQUE NOT NULL, Val TEXT );
INSERT INTO TestTable (Key,Val) VALUES ('Fruit', 'banana') ON CONFLICT( Key ) DO UPDATE SET Val=EXCLUDED.Val;
INSERT INTO TestTable (Key,Val) VALUES ('Fruit', 'apple') ON CONFLICT( Key ) DO UPDATE SET Val=EXCLUDED.Val;
INSERT INTO TestTable (Key,Val) VALUES ('Fruit', 'peach') ON CONFLICT( Key ) DO UPDATE SET Val=EXCLUDED.Val;
INSERT INTO TestTable (Key,Val) VALUES ('Animal', 'horse') ON CONFLICT( Key ) DO UPDATE SET Val=EXCLUDED.Val;
SELECT * FROM TestTable;
id Key Val
1 Fruit peach
4 Animal horse
In this case, each conflict during the Fruit update has bumped the SERIAL value, even though it wasn't creating any new record in TestTable.
Now this is the workaround I'm currently working with. If anyone knows how to concatenate the table name onto 'NEW.id', I'd love to hear that, as I like to name my id columns idTablename for consistency.
CREATE OR REPLACE FUNCTION IncrementSerial()
RETURNS trigger AS $fn$
BEGIN
EXECUTE format('SELECT COALESCE( MAX( id ), 0 ) + 1 FROM %I.%I;',TG_TABLE_SCHEMA,TG_TABLE_NAME) INTO NEW.id;
RETURN NEW;
END
$fn$ LANGUAGE 'plpgsql'
CREATE TABLE TestTable ( id INTEGER PRIMARY KEY NOT NULL, Key TEXT UNIQUE NOT NULL, Val TEXT );
CREATE TRIGGER trgIncrementSerial
BEFORE INSERT ON TestTable
FOR EACH ROW
EXECUTE PROCEDURE IncrementSerial()
INSERT INTO TestTable (Key,Val) VALUES ('Fruit', 'banana') ON CONFLICT( Key ) DO UPDATE SET Val=EXCLUDED.Val;
INSERT INTO TestTable (Key,Val) VALUES ('Fruit', 'apple') ON CONFLICT( Key ) DO UPDATE SET Val=EXCLUDED.Val;
INSERT INTO TestTable (Key,Val) VALUES ('Fruit', 'peach') ON CONFLICT( Key ) DO UPDATE SET Val=EXCLUDED.Val;
INSERT INTO TestTable (Key,Val) VALUES ('Animal', 'horse') ON CONFLICT( Key ) DO UPDATE SET Val=EXCLUDED.Val;
SELECT * FROM TestTable;
id Key Val
1 Fruit peach
2 Animal horse
As you can see, the SERIAL id is now just using the next highest id number, which is ideal for most of my use cases.
Obviously this is going to be a problem if the id key must always be unique, as removal of the last record will free up that id. If that's not a problem (ie. where the id is just used for references and cascades) then this might be a good solution for you.