How to efficiently insert tree-like data structure into postgres - sql

Question

Essentially, I want to efficiently store a tree-like data structure in a table with Postgres. Each row has an ID (auto-generated upon insert), a parent ID (referencing another row in the same table, possibly null), and some additional metadata. All of that data comes in at once, so I'm trying to store it all at once as efficiently as possible.
My current thought is to group all the data by which level of the tree they're at, and batch insert one level at a time. That way I can set parent IDs using the IDs generated from the previous level's inserts. This way the amount of batches is correlated with the number of levels in the tree.
This is probably "good enough", but I'm wondering if there's a better way to do this kind of thing? It still seems like a lot of back and forth and unnecessary logic to me, when I have the whole tree of data already in memory and structured correctly.

Answer 1

Let me show how I would do it if I had some information on who is whose child record.
In my case, I use a staging table containing the info as it comes from the source. The records have a char based primary key id, and a self-referencing,nullable, foreign key boss_id .
Here goes:
-- the input table with "business identifiers".
DROP TABLE IF EXISTS rec_input;
CREATE TABLE rec_input (
id CHAR(4)
, first_name VARCHAR(32)
, last_name VARCHAR(32)
, boss_id CHAR(4)
)
;
-- some data for it ...
INSERT INTO rec_input(id,first_name,last_name,boss_id)
SELECT 'A01','Arthur','Dent' ,NULL
UNION ALL SELECT 'A02','Ford','Prefect' ,'A01'
UNION ALL SELECT 'A03','Zaphod','Beeblebrox' ,'A01'
UNION ALL SELECT 'A04','Tricia','McMillan' ,'A01'
UNION ALL SELECT 'A05','Gag','Halfrunt' ,'A02'
UNION ALL SELECT 'A06','Prostetnic Vogon','Jeltz','A02'
UNION ALL SELECT 'A07','Lionel','Prosser' ,'A04'
UNION ALL SELECT 'A08','Benji','Mouse' ,'A04'
UNION ALL SELECT 'A09','Frankie','Mouse' ,'A04'
UNION ALL SELECT 'A10','Svlad','Cjelli' ,'A03'
;
-- create a lookup table. The surrogate key is created here.
DROP TABLE IF EXISTS lookup_help;
CREATE TABLE lookup_help (
sk SERIAL NOT NULL -- < here is the surrogate auto increment key
, id CHAR(3)
);
-- fill the lookup table
INSERT INTO lookup_help(id)
SELECT id FROM rec_input;
-- test query
SELECT * FROM lookup_help;
-- this is the target table, with auto increment
-- and matching surrogate foreign key.
DROP TABLE IF EXISTS rec;
CREATE TABLE rec (
sk INTEGER NOT NULL -- surrogate key
, id CHAR(4). -- "business id"
, first_name VARCHAR(32)
, last_name VARCHAR(32)
, boss_id CHAR(4). -- "business foreign key", not needed really
, boss_sk INTEGER. -- internal foreign key
)
;
INSERT INTO rec
SELECT
l.sk -- from lookup table, inner joined
, i.id -- from input table
, i.first_name
, i.last_name
, i.boss_id
, b.sk -- from lookup table, left outer joined
FROM rec_input i
JOIN lookup_help l USING(id) -- for the main sk
LEFT JOIN lookup_help b ON i.boss_id=b.id -- for the foreign sk
;
-- test query
SELECT * FROM rec;
-- out sk | id | first_name | last_name | boss_id | boss_sk
-- out ----+------+------------------+------------+---------+---------
-- out 2 | A02 | Ford | Prefect | A01 | 1
-- out 3 | A03 | Zaphod | Beeblebrox | A01 | 1
-- out 4 | A04 | Tricia | McMillan | A01 | 1
-- out 6 | A06 | Prostetnic Vogon | Jeltz | A02 | 2
-- out 5 | A05 | Gag | Halfrunt | A02 | 2
-- out 10 | A10 | Svlad | Cjelli | A03 | 3
-- out 7 | A07 | Lionel | Prosser | A04 | 4
-- out 8 | A08 | Benji | Mouse | A04 | 4
-- out 9 | A09 | Frankie | Mouse | A04 | 4
-- out 1 | A01 | Arthur | Dent | |
-- out (10 rows)

Answer 2

Perhaps with your use case, you could try NoSql at the moment, querying such data would be far efficient and faster. Maybe give it a shot.
For development you've options like Apache CouchDB, redis, etc.