Keep a relation map in Objection.js while removing the table - sql

I'm developing a reddit-like site where votes are stored per-user (instead of per-post). Here's my relevant schema:
content
id | author_id | title | text
---|-----------|-------------|---
1 | 1 (adam) | First Post | This is a test post by adam
vote: All the votes ever voted by anyone on any post
id | voter_id | content_id | category_id
---|-------------|------------------|------------
1 | 1 (adam) | 1 ("First Post") | 1 (upvote)
2 | 2 (bob) | 1 ("First Post") | 1 (upvote)
vote_count: Current tally ("count") of total votes received by a post by all users
id | content_id | category_id | count
---|------------------|--------------|-------
1 | 1 ("First Post") | 1 (upvote) | 2
I've defined a voteCount relation in Objection.js model for the content table:
class Content extends Model {
static tableName = 'content';
static relationMappings = {
voteCount: {
relation: Model.HasManyRelation,
modelClass: VoteCount,
join: {
from: 'content.id',
to: 'vote_count.content_id'
}
}
}
}
But I recently (learned and) decided that I don't need to keep (and update) a separate vote_count table, when in fact I can just query the vote table and essentially get the same table as a result:
SELECT content_id
, category_id
, COUNT(*) AS count
FROM vote
GROUP
BY content_id
, category_id
So now I wanna get rid of the vote_count table entirely.
But it seems that would break my voteCount relation since there won't be a VoteCount model (not shown here but it's the corresponding the model for the vote_count table) no more either. (Right?)
How do I keep voteCount relation while getting rid of vote_count table (and thus VoteCount model with it)?
Is there a way to somehow specify in the relation that instead of looking at a concrete table, it should look at the result of a query? Or is it possible to define a model class for the same?
My underlying database in PostgreSQL if that helps.

Thanks to #Belayer. Views were exactly the solution to this problem.
Objection.js supports using views (instead of table) in a Model class, so all I had to do was create a view based on the above query.
I'm also using Knex's migration strategy to create/version my database, and although it doesn't (yet) support creating views out of the box, I found you can just use raw queries:
module.exports.up = async function(knex) {
await knex.raw(`
CREATE OR REPLACE VIEW "vote_count" AS (
SELECT content_id
, category_id
, COUNT(*) AS count
FROM vote
GROUP
BY content_id
, category_id
)
`);
};
module.exports.down = async function(knex) {
await knex.raw('DROP VIEW "vote_count";');
};
The above migration step replaces my table vote_count for the equivalent view, and the Objection.js Model class for it (VoteCount) worked as usual without needing any change, and so did the relation voteCount on the Content class.

Related

SQL / typeORM query to get a DM conversation in a chat web application

I am pretty new to SQL and typeORM, and I especially have troubles when it comes to many to many relationships.
I am working on a chat application. I have two tables connected by a many to many to many relationship.
========= =========
| Room | | User |
========= =========
| id | -< | id |
--------- / ---------
| type | / | ... |
--------- / ---------
| User[]| >--
---------
| ... |
---------
Room.type can be of value dm, private or public
I also have a join table in between the two tables, but it is handled by typeORM so I don't really care about it.
What I want to do is to query the Room table and get all the rooms of type dm where the user are exactly user1_id and user2_id (no more, no less) but I really don't know how to formulate this query. Something like:
SELECT * FROM 'Room'
WHERE type IS 'dm'
AND WHERE 'user1_id' IN 'User.id'
AND WHERE 'user2_id' IN 'User.id'
AND WHERE LENGTH(Room.User) = 2
I am posting this with SQL but eventually I will translate it with typeORM syntax which should not be a problem.
Thanks.
Since there's a many to many relation between Room and User and as the model looks like Room has an array of User, there'd be a join table in the database named room_users_user. Now you want all rooms where only user1_id and user2_id are participants. The corresponding SQL query should be:
SELECT rooms.id from rooms
LEFT JOIN room_users_user ON rooms.id = room_users_user.room_id
LEFT JOIN users ON room_users_user.user_id = users.id
WHERE rooms.type = 'dm' AND users.id IN (2, 4)
GROUP BY rooms.id
HAVING COUNT(users.id) = 2;
which can be written in typeorm like this:
const roomIds = await roomRepository.createQueryBuilder('room')
.leftJoinAndSelect('room.users', 'users')
.select('room.id', 'roomId')
.where('room.type = :type', { type: 'dm' })
.andWhere('users.id in (:...userId)', { userId: [user1_id, user2_id] })
.groupBy('room.id')
.having('count(users.id) = 2')
.getRawMany();
A DB Fiddle for better understanding.

My SQL query is taking too long, is there another approach?

I want to store 3D vector images in my MariaDB, but I'm finding that retrieving the data is taking far too long to be practical.
I have a few tables:
a points table containing the x,y and z coordinates plus the entity id,
an entity table containing a unique id, an entity type (text, line, polyline,etc) other common attributes such as colour and linetype,
and some auxiliary tables containing additional values like text, text height, line thicknesses and flags split into separate tables based on field type (varchar, int or float).
I am accessing the data through PHP as follows:
if($result = mysqli_query($conn, "SELECT entityID,X,Y,Z FROM dwgpoints WHERE drawing=".$DrawingID." AND blockID=".$blockID.";"))
{
$previous_eID=0;
while($row = mysqli_fetch_array($result))
{
$eID=$row['entityID'];
if($previous_eID!=$eID)
{
if($previous_eID)// confirm it's not zero
renderEntity($image_handle,$DrawingID,$previous_eID,$etype,$colour,$ltype,$points, $transformation, $clip);
$previous_eID=$eID;
if($eResult=mysqli_query($conn,"SELECT colour,ltype,etype FROM entity WHERE drawing=".$ID." AND eID=".$eID.";")){
$erow=mysqli_fetch_assoc($eResult);
$colour=$erow['colour'];
$ltype=$erow['ltype'];
$etype=$erow['etype'];
$points=[[$row['X'],$row['Y'],$row['Z']]];
}
}else{
$points[]=[$row['X'],$row['Y'],$row['Z']];
}
}
}
This process is taking up to ten minutes, but I know that Openstreetmaps, for example, renders tiles from similar amounts of data.
The results of the EXPLAIN directive is as follows:
MariaDB [wptest_11]> EXPLAIN SELECT entityID,X,Y,Z FROM dwgpoints WHERE drawing=2 AND blockID=-1;
+------+-------------+-----------+------+---------------+--------+---------+-------------+-------+-----------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+------+-------------+-----------+------+---------------+--------+---------+-------------+-------+-----------------------+
| 1 | SIMPLE | dwgpoints | ref | idx_id | idx_id | 9 | const,const | 24939 | Using index condition |
+------+-------------+-----------+------+---------------+--------+---------+-------------+-------+-----------------------+
Is it possible to streamline my data searches to make the process time manageable? I'm on a cheap VPS, so there may be hardware performance issues, in which case would an upgrade make much difference? Or do I need to rethink my approach?
Any advice would be most welcome.
For reference, I have run millions of records on a Raspberry Pi3 with better performance. Hardware helps, but I would first look at:
Minimizing your calls, if it makes sense for your application
Add some indexes on fields in your WHERE clause to improving performance
Single Query
Adjust the following query to fit your needs. Then see what the timing is. For 26k results, you should be well under a second.
SELECT
a.entityID
, a.X
, a.Y
, a.Z
, b.colour
, b.ltype
, b.etype
, b.drawing
FROM dwgpoints a
LEFT JOIN entity b
ON b.eID = a.entityID
AND b.drawing = a.drawing
WHERE
a.drawing = ".$ID."
AND a.blockID = ".$blockID.
;
Indexes
Create some indexes:
The combination of drawing and blockID for the dwgpoints table.
The combination of drawing and eID for the entity table.
Re-run the above query, there should be an improvement.
I would also look at adding a foreign key between the tables, which will help improve data integrity of your database.

Laravel: Adding multiple count results of sub queries

I wanted to add the count result of more than one queries that belong to different tables.
I am using the below problem as a reference to my actual problem because this problem has already a solution (How do I add two count(*) results together on two different tables?) but I am facing problem in implementing the solution in laravel.
I have two tables: Toys and Games.
+--------------------+------------------+
| Field | Type |
+--------------------+------------------+
| toy_id | int(10) unsigned |
| little_kid_id | int(10) unsigned |
+--------------------+------------------+
+--------------------+------------------+
| Field | Type |
+--------------------+------------------+
| game_id | int(10) unsigned |
| little_kid_id | int(10) unsigned |
+--------------------+------------------+
A little kid can have multiple toys. A little kid can be participating in multiple games at once.
I want a query that will give me the total number of toys + games that a little_kid is involved with.
Basically, I want the sum of these two queries:
SELECT COUNT(*) FROM Toys WHERE little_kid_id = 900;
SELECT COUNT(*) from Games WHERE little_kid_id = 900
The above problem has the following accepted answer
SELECT
(SELECT COUNT(*) FROM Toys WHERE little_kid_id = 900)+
(SELECT COUNT(*) from Games WHERE little_kid_id = 900)
AS SumCount
I wanted to implement the above solution in Laravel.
I have tried the following method but to no avail. It gives syntax error.
$sub=DB::tabel('toys')->select(DB::raw('count(*) as total'))
->where('little_kid_id',$id);
$sub1=DB::tabel('games')->select(DB::raw('count(*) as total'))
->where('little_kid_id',$id);
$result = DB::table( DB::raw("( ({$sub->toSql()})+({$sub1->toSql()}) ) as
total_count") )
->mergeBindings($sub)
->mergeBindings($sub1)
->select('total_count.total')
->get();
I have also tried this method. It works but gives collection but I need an integer value of total count
$result=$sub->unionAll($sub1);
dd($result->get());
In short I wanted to perform the accepted solution of that problem (How do I add two count(*) results together on two different tables?) in laravel.
You can use those codes :
$q = DB::tabel('toys')->select('toy_id','little_kid_id')->where('little_kid_id',900);
$q1 = DB::tabel('games')->select('game_id','little_kid_id')->where('little_kid_id',900)
$data = $q->union($q1)->count();
Don't forget Union require the tables must have the same columns so that I select the columns if your columns will not match each other then don't touch the select statement otherwise feel free to remove the codes
The second way is :
DB::table(DB::raw('(SELECT COUNT(*) as c FROM Toys WHERE little_kid_id = 900) t,(SELECT COUNT(*) as c1 from Games WHERE
little_kid_id = 900) t2'))->selectRaw('t.c+t2.c1 as
SumCount')->toSql(); //change toSql() to get() if you want to get
datas instead of sql code
You can try this. for more information look at https://laravel.com/docs/5.8/queries#raw-expressions. if you want to get request parameters by any user then be confirmed to prevent SQL Injection

django complex datamodel

I am creating a small Django project which show stats collected from twitter data
for example my tables are
hashDetails
---------------------------------------------
id hashname tweetPosted trendDate userid
---------------------------------------------
1 #abc 44 2-2-2016 #xyz
2 #abc 55 2-2-2016 #qwer
3 #xcs 55 3-2-2016 #qwer
4 #xcs 55 4-2-2016 #qwer
---------------------------------------------
userDetails
----------------------------------------------
id userid profileImage profileImage
----------------------------------------------
1 #xyz image2.jpg www.abc.com
2 #qwer image3.jpg www.xadf.com
----------------------------------------------
for this if i create models.py
class userDetails(models.Model):
userid= models.CharField(max_length=30)
profileImage= models.CharField(max_length=30)
profileImage= models.CharField(max_length=30)
class hashDetails(models.Model):
hashname= models.CharField(max_length=30)
tweetPosted= models.IntegerField()
trendDate= models.DateTimeField()
userid = models.ForeignKey(userDetails, to_field ='userid')
but i don't wanna make userid unique cause
i want something like i can enter data in both table manually
and when i query in my view it will search result from both table
example
if i want all trends by #xyz
or if i want list of all users who did #abc trend
or if i want result of all trends in specific date
in short i want both table to behave like one
I can't use userid as unique my daily data will be about 20MB so you can assume its difficult to find ids
I found one solution of my problem and its working for me
i just create normal 2 model without foreignkey or any relation
and define my function in views.py
and got my result what i want
def teamdetail(request,test_id):
hashd = hashDetails.objects.get(hashname=test_id)
userd= userDetails.objects.all()
context = {'hashinfo': hashd, 'username':userd}
return render(request,'test/hashDetails.html',context)

Sql recursion without recursion

I have four tables
create table entities{
integer id;
string name;
}
create table users{
integer id;//fk to entities
string email;
}
create table groups{
integer id;//fk to entities
}
create table group_members{
integer group_id; //fk to group
integer entity_id;//fk to entity
}
I want to make a query that returns all groups where a user belongs, directly or indirectly. The obvious solution is to make a recursion at the application level. I’m wondering what changes can I make to my data model to decrease the database access and as a result have a better performance.
In Oracle:
SELECT group_id
FROM group_members
START WITH
entity_id = :user_id
CONNECT BY
entity_id = PRIOR group_id
In SQL Server:
WITH q AS
(
SELECT group_id, entity_id
FROM group_members
WHERE entity_id = #user_id
UNION ALL
SELECT gm.group_id, gm.entity_id
FROM group_members gm
JOIN q
ON gm.entity_id = q.group_id
)
SELECT group_id
FROM q
In PostgreSQL 8.4:
WITH RECURSIVE
q AS
(
SELECT group_id, entity_id
FROM group_members
WHERE entity_id = #user_id
UNION ALL
SELECT gm.group_id, gm.entity_id
FROM group_members gm
JOIN q
ON gm.entity_id = q.group_id
)
SELECT group_id
FROM q
In PostgreSQL 8.3 and below:
CREATE OR REPLACE FUNCTION fn_group_members(INT)
RETURNS SETOF group_members
AS
$$
SELECT group_members
FROM group_members
WHERE entity_id = $1
UNION ALL
SELECT fn_group_members(group_members.group_id)
FROM group_members
WHERE entity_id = $1;
$$
LANGUAGE 'sql';
SELECT group_id
FROM group_members(:myuser) gm
There are ways of avoiding recursion in tree hierarchy queries (in opposition to what people have said here).
The one I've used most is Nested Sets.
As with all life and technical decisions, however, there are trade offs to be made. Nested Sets are often slower to update but much faster to query. There are clever and complicated ways of improving the speed of updating the hierarchy, but there's another trade-off; performance vs code complexity.
A simple example of a nested set...
Tree View:
-Electronics
|
|-Televisions
| |
| |-Tube
| |-LCD
| |-Plasma
|
|-Portable Electronics
|
|-MP3 Players
| |
| |-Flash
|
|-CD Players
|-2 Way Radios
Nested Set Representation
+-------------+----------------------+-----+-----+
| category_id | name | lft | rgt |
+-------------+----------------------+-----+-----+
| 1 | ELECTRONICS | 1 | 20 |
| 2 | TELEVISIONS | 2 | 9 |
| 3 | TUBE | 3 | 4 |
| 4 | LCD | 5 | 6 |
| 5 | PLASMA | 7 | 8 |
| 6 | PORTABLE ELECTRONICS | 10 | 19 |
| 7 | MP3 PLAYERS | 11 | 14 |
| 8 | FLASH | 12 | 13 |
| 9 | CD PLAYERS | 15 | 16 |
| 10 | 2 WAY RADIOS | 17 | 18 |
+-------------+----------------------+-----+-----+
You'll want to read the article I linked to understand this fully, but I'll try to give a short explanation.
An item is a member of another item if (the child's "lft" (Left) value is greater than the parent's "ltf" value) AND (the child's "rgt" value is less than the parent's "rgt" value)
"Flash" is therfore a member of "MP3 PLAYERS", "Portable Electronics" and "Electronics"
Or, conversley, the members of "Portable Electronics" are:
- MP3 Players
- Flash
- CD Players
- 2 Way Radios
Joe Celko has an entire book on "Trees and Hierarchies in SQL". There are more options than you think, but lots of trade off's to make.
Note: Never say something can't be done, some mofo will turn up to show you that in can.
Can you clarify the difference between an entity and a user? Otherwise, your tables look OK. You are making an assumption that there is a many-to-many relationship between groups and entities.
In any case, with standard SQL use this query:
SELECT name, group_id
FROM entities JOIN group_members ON entities.id = group_members.entity_id;
This will give you a list of names and group_ids, one pair per line. If an entity is a member of multiple groups, the entity will be listed several times.
If you're wondering why there's no JOIN to the groups table, it's because there's no data from the groups table that isn't already in the group_members table. If you included, say, a group name in the groups table, and you wanted that group name to be shown, then you'd have to join with groups, too.
Some SQL variants have commands related to reporting. They would allow you to list multiple groups on the same line as a single entity. But it's not standard and wouldn't work across all platforms.
If you want a truly theoretically infinite level of nesting, then recursion is the only option, which precludes any sane version of SQL. If you're willing to limit it, then there are a number of other options.
Check out this question.
You can do the following:
Use the START WITH / CONNECT BY PRIOR constructs.
Create a PL/SQL function.
I don't think there is a need for recursion here as the solution posted by barry-brown seems adequate. If you need a group to be able to be a member of a group, then the tree traversal method offered by Dems works well. Inserts, deletes and updates are pretty straightforward with this scheme, and retrieving the entire hierarchy is accomplished with a single select.
I would suggest including a parent_id field in your group_members table (assuming that is the point at which your recursive relationship occurs). In a navigation editor I've created a nodes table like so:
tbl_nodes
----------
node_id
parent_id
left
right
level
...
My editor creates hierarchically-related objects from a C# node class
class node {
public int NodeID { get; set; }
public Node Parent { get; set; }
public int Left { get; set; }
public int Right { get; set; }
public Dictionary<int,Node> Nodes { get; set; }
public int Level {
get {
return (Parent!=null) ? Parent.Level+1 : 1;
}
}
}
The Nodes property contains a list of child nodes. When the business layer loads the hierarchy, it rectifies the parent/child relationships. When the nav editor saves, I recursively set the left and right property values, then save to the database. That lets me get the data out in the correct order meaning I can set parent/child references during retrieval instead of having to make a second pass. Also means that anything else that needs to display the hierarchy ( say, a report) can easily get the node list out in the correct order.
Without a parent_id field, you can retrieve a breadcrumb trail to the current node with
select n1.*
from nodes n1, nodes n2
where d1.lft <= d2.lft and d1.rgt >= d2.rgt
and d2.id = #id
order by lft;
where #id is the id of the node you're interested in.
Pretty obvious stuff, really, but it applies to items such as nested group membership that might not be obvious, and as others have said eliminates the need to slow recursive SQL.