How to set the explicit order for child table rows for one-to-many SQL relation? - sql

Question

Imagine a database with two tables, lists (with id and name) and items (with id, list_id, which is a foreign key linking to lists.id, and name) and the application with ORM and the corresponding models.
A task: have a way in the application to create/edit/view the list and the items inside it (that should be pretty easy), but also saving the order of the items within one list and allowing to reorder the items within one list (so, a user creates the items list, then swaps two items, then when displaying the list, the items order should be preserved) or deleting items.
What is the best way to implement it, database-wise? Which db structure should I use for it?
I see these ways of solving it:
not using the external table for items, but storing everything in a list document (as a postgres jsonb column for example) - can work but I suppose that's not RDBMS way to do it and if the user would want to update the single item, the whole list object would need to be updated
having a position field in items table and adding a way to manage the position in the API - can work, but it's quite complicated (like, handling the cases where the position is the same for some items, handling swapping items, handling items deletions and having to decrease the position of all the items coming after the deleted one etc.)
Is there a simple way of implementing it? Like the one used in production by some big companies? I'm really curious about how such cases are handled in real life.
This is more theoretical question, so no code samples here (except for the db structure).

Answer 1

This is a good question, which as far as I know doesn't have any simple answers. I once came up with a solution for a high volume photo sharing site using an item table with columns list_id and position as you describe. The key to performance was to minimize renumbering as this database had millions of photos (and more than 2^32 likes).
The only operation was to move a single item to another point in the list (before or after another item in the list). This would work by first assigning positions with large steps, e.g. 1000, 2000, 3000. Whenever an item is moved between two others the average is used, e.g. move from pos=3000 to 1500. Eventually you can try to move an item between two items that have consecutive position numbers. Then you choose to renumber items either above or below depending on which way requires fewer updates (e.g. if there were a run of consecutive positions). This was done using RANK and #vars as I recall on MySQL 5.7.
This did work well resolving a problem where there was intermittent unavailability in production due to massive renumberings that were occurring before when consecutive positions were used.
I was able to dig up a couple of the queries (that was meant to go into a blog post ages ago). Turns out this was MySQL before RANK() was a thing which is why the #shuffle_rank variable was used. The + 0 (and the + 1) is because this is the actual SQL sent to the query but it was generated in code. This is to find the first gap below (greater than) position 120533287:
SELECT shuffle_rank, position
FROM (SELECT #shuffle_rank := #shuffle_rank + 1 AS shuffle_rank, position
FROM `gallery_items`
JOIN (SELECT #shuffle_rank := 0) initialize_rank_var
WHERE `gallery_items`.`gallery_id` = 14103882 AND (position >= 120533287)
ORDER BY position ASC) positionable_items
WHERE ABS(120533287 - position) >= shuffle_rank + 0 LIMIT 1
Here's the update query after the above query and supporting code decided that 3 rows need to be updated to make a gap. The + 1 here may be larger if renumbering with some gap if there's room.
UPDATE `gallery_items`
SET position = -222 + (#shuffle_rank := #shuffle_rank + 1)
WHERE `gallery_items`.`gallery_id` = 24669422
AND (position >= -222)
AND ((SELECT #shuffle_rank := 0) = 0)
ORDER BY position ASC
LIMIT 3
Note that this pair of actual queries aren't for the same operation seeing as they have different gallery_id values (aka list_id).