Recursive query to find previous related row meeting criteria - sql

Question

I have a database full of messages from various chatbots. The chatbots all
follow decision tree format and ultimately are questions presented with choices
to which the user responds.
The bot may send a message (Hello would you like A or B?) which has options
attached, A and B for example. The user responds B. Both of these messages are
recorded and the previous message id attached.
id
message
options
previous_id
1
Hello would you like A or B?
A,B
2
A
1
The structure of these conversations is not fixed. There may be various forms
of message flow. The above is a simplistic example of how the messages are
chained together. For example
// text question on same message as options, with preceding unrelated messages
Hello -> My name is Ben the bot. -> How are you today? (good, bad) -> [good]
// text question not on same message as options
Pick your favourite colour -> {picture of blue and red} (blue, red) -> [blue]
// no question just option prompt - here precending text wasn't a question
[red] -> (ferrari, lamborghini) -> [ferrari]
-> denotes separation of messages
[] denotes reply to bot from user
() denotes options attached to messages
{} denotes attachments
What I am trying to get from this data is a row for every question with its
corresponding answer. The problem i'm facing is the (presumable) recursion i'd
have to use to retrieve the previous message each time until it met criteria
indicating it's gone back far enough for that particular answer in the chain of
messages.
In theory what I am trying to achieve is
Find all answers to questions
From those results look at the previous message
2a. If previous message has text and is not an answer itself then use said text and stop recursing
2b. Else move onto the next previous message until the criteria is met.
Return rows containing answer/response, with question and other columns from question row (id, timestamp for example)
This would leave me with lots of rows containing a message and a response
in the following dataset for example,
id
message
other
previous_id
1
Hello would you like A or B?
2
B
1
3
Hello would you like A or B?
4
A
3
5
Hello would you like A or B?
6
B
5
7
A is a great answer. C or D?
4
8
D
7
9
Green or red?
10
image
9
11
Red
10
I'd hope to end up with
id
message
response
1
Hello would you like A or B?
B
3
Hello would you like A or B?
A
5
Hello would you like A or B?
B
7
A is a great answer. C or D?
D
8
Green or red?
Red
I have made a (somewhat) simplified version of some sample data which is at the bottom of this question for reference/use.
It uses the following structure
WITH data ( id, message, node, options, previous, attachment) AS ()
Answers can be found with select where node is null so I assumed that is the
best starting point and I can work backwards towards the question. previous
and options are json columns because that's how they are in the real data so
I left them as they were.
I have tried various means by which to get the data as I wanted but I haven't managed the recursion/unknown number of levels bit.
For example, this attempt can dig two levels deep but I couldn't coalesce the
id of the message i found because obviously both have non null values.
select COALESCE(d2.message, d3.message) as question, d.message as answer
-- select COALESCE(d2.message, d2.attachment, d3.message, d3.attachment) as question, d.message as answer
from data as d
left join data as d2 on (d.previous->>'id')::int = d2.id
left join data as d3 on (d2.previous->>'id')::int = d3.id
where d.previous->>'node' in (
SELECT node from data where options is not null group by node
)
I believe this answer https://dba.stackexchange.com/a/215125/4660 may be the
path to what I need but I've thus far been unable to get it to run as I'd like.
I think this would allow me to replace the two left joins in my above example
with say a recursive union which i can use conditions on the on clause to stop
it at the right point. Hopefully this sounds like it might be along the right
lines and someone can point me in the right direction. Something like the below
perhaps?
WITH data (
id,
message,
node,
options,
previous,
attachment
) AS (
VALUES ...
), RecursiveTable as (
select * from data d where node is null # all answers?
union all
select * from RecursiveTable where ??
)
select * from RecursiveTable
--
Basic sample dataset
WITH data (
id,
message,
node,
options,
previous,
attachment
) AS (
VALUES
-- QUESTION TYPE 1
-- pineapple questions
(1, 'Pineapple on pizza?', 'pineapple', '["Yes","No"]'::json, null::json, null),
(2, 'Pineapple on pizza?', 'pineapple', '["Yes","No"]'::json, null::json, null),
(3, 'Pineapple on pizza?', 'pineapple', '["Yes","No"]'::json, null::json, null),
(4, 'Pineapple on pizza?', 'pineapple', '["Yes","No"]'::json, null::json, null),
(5, 'Pineapple on pizza?', 'pineapple', '["Yes","No"]'::json, null::json, null),
-- pineapple answers
(6, 'No', null, null, '{"id": 1, "node": "pineapple"}'::json, null),
(7, 'Yes', null, null, '{"id": 2, "node": "pineapple"}'::json, null),
(8, 'No', null, null, '{"id": 3, "node": "pineapple"}'::json, null),
(9, 'Yes', null, null, '{"id": 4, "node": "pineapple"}'::json, null),
(10, 'No', null, null, '{"id": 5, "node": "pineapple"}'::json, null),
-- ----------------------------
-- QUESTION TYPE 2 - Previous message, then question with text + options followed by answer
--- previous messages to stuffed crust questions (we don't care about
--these but they're here to ensure we aren't accidentally getting them
--as the question in results)
(11, 'Hello', 'hello_pre_stuffed_crust', null, null::json, null),
(12, 'Hello', 'hello_pre_stuffed_crust', null, null::json, null),
(13, 'Hello', 'hello_pre_stuffed_crust', null, null::json, null),
-- stuffed crust questions
(14, 'Stuffed crust?', 'stuffed_crust', '["Crunchy crust","More cheese!"]'::json, '{"id": 11, "node": "hello_pre_stuffed_crust"}'::json, null),
(15, 'Stuffed crust?', 'stuffed_crust', '["Crunchy crust","More cheese!"]'::json, '{"id": 12, "node": "hello_pre_stuffed_crust"}'::json, null),
(16, 'Stuffed crust?', 'stuffed_crust', '["Crunchy crust","More cheese!"]'::json, '{"id": 13, "node": "hello_pre_stuffed_crust"}'::json, null),
-- stuffed crust answers
(17, 'More cheese!', null, null, '{"id": 14, "node": "stuffed_crust"}'::json, null),
(18, 'Crunchy crust', null, null, '{"id": 15, "node": "stuffed_crust"}'::json, null),
(19, 'Crunchy crust', null, null, '{"id": 16, "node": "stuffed_crust"}'::json, null),
-- ----------------------------
-- QUESTION TYPE 3
-- two part question, no text with options only image, should get text from previous
-- part 1
(20, 'What do you think of this pizza?', 'check_this_image', null, null::json, null),
(21, 'What do you think of this pizza?', 'check_this_image', null, null::json, null),
(22, 'What do you think of this pizza?', 'check_this_image', null, null::json, null),
-- part two
(23, null, 'image', '["Looks amazing!","Not my cup of tea"]'::json, '{"id": 20, "node": "check_this_image"}'::json, 'https://images.unsplash.com/photo-1544982503-9f984c14501a'),
(24, null, 'image', '["Looks amazing!","Not my cup of tea"]'::json, '{"id": 21, "node": "check_this_image"}'::json, 'https://images.unsplash.com/photo-1544982503-9f984c14501a'),
(25, null, 'image', '["Looks amazing!","Not my cup of tea"]'::json, '{"id": 22, "node": "check_this_image"}'::json, 'https://images.unsplash.com/photo-1544982503-9f984c14501a'),
-- two part answers
(26, 'Looks amazing!', null, null, '{"id": 23, "node": "image"}'::json, null),
(27, 'Not my cup of tea', null, null, '{"id": 24, "node": "image"}'::json, null),
(28, 'Looks amazing!', null, null, '{"id": 25, "node": "image"}'::json, null),
-- ----------------------------
-- QUESTION TYPE 4
-- no text, just options straight after responding to something else - options for text value would be options, or image
-- directly after question 3 was answered, previous message was user message - but we don't have text here - just an image and options
(29, null, 'which_brand', '["Dominos","Papa Johns"]'::json, '{"id": 27}'::json, 'https://peakstudentmediadotcom.files.wordpress.com/2018/11/vs.jpg'),
(30, null, 'which_brand', '["Dominos","Papa Johns"]'::json, '{"id": 28}'::json, 'https://peakstudentmediadotcom.files.wordpress.com/2018/11/vs.jpg'),
(31, null, 'which_brand', '["Dominos","Papa Johns"]'::json, '{"id": 29}'::json, 'https://peakstudentmediadotcom.files.wordpress.com/2018/11/vs.jpg')
)
SELECT * from data

Answer 1

You can use WIT HRECURSIVE to achieve your goal. You just need to specify when to stop the recursion and find a way to select only those records, where the recursion did not produce any additional rows for.
Have a look here:
WITH RECURSIVE comp (
id, message, node, options, previous, attachment,
id2, message2, node2, options2, previous2, attachment2,
rec_depth
) AS (
SELECT
t.id, t.message, t.node, t.options, t.previous, t.attachment,
null::integer AS id2, null::text AS message2, null::text AS node2, null::json AS options2, null::json AS previous2, null::text AS attachment2,
0
FROM data t
WHERE t.node IS NULL
UNION ALL
SELECT
c.id, c.message, c.node, c.options, c.previous, c.attachment,
prev.id, prev.message, prev.node, prev.options, prev.previous, prev.attachment,
c.rec_depth + 1
FROM comp c
INNER JOIN data prev ON prev.id = ((COALESCE(c.previous2, c.previous))->>'id')::int
WHERE prev.node IS NOT NULL -- do not reach back to the next answer
AND c.message2 IS NULL -- do not reach back beyond a message with text (the question text)
), data (id, message, node, options, previous, attachment) AS (
VALUES [...]
) SELECT
c.id2 AS question_id, c.id AS answer_id
FROM comp c
WHERE
NOT EXISTS(
SELECT 1
FROM comp c2
WHERE c2.id = c.id
AND c2.rec_depth > c.rec_depth
)
comp holds before the recursion only the "answers" (this is the part above UNION ALL). Then, in the first recursion step, they are joined with the predecesors. In the second step, another new record is created per answer-predecessor pair, where the predecessor replaces itself with its predecessor. This is done, until the "base-condition" (the joined partner is a record with message aka question text or the next partner is a record without node aka an answer) is reached (this means until no new records get created).
As we also compute the recursion depth (rec_depth) of each row, we can finally check that we use only those records generated per answer with the maximal recursion depth.
The second WITH statement can and should of course be removed and you should reference your real table in the WITH RECURSIVE part.
I chose to only select the ids of the answer and the corresponding question, but the WITH RECURSIVE is already built in a way, that you can use all of the columns.
Further reading in the docs:
https://www.postgresql.org/docs/13/sql-select.html#SQL-WITH
https://www.postgresql.org/docs/13/queries-with.html