Retrieve data on many tables - Laravel - sql

Good morning,
Here is my model :
A Chretien can have many Poste. A Poste can belog to many Chretien.
A Poste belongs to many Departement. A Departement has many Poste.
CHRETIEN------------------------POSTE---------------------------DEPARTEMENT
0..* 0..* 1..** 0.. *
How can I retrieve the model like this?
John DOE
---------------------------------------
|**Postes** | **Departements** |
---------------------------------------
|Pianist | Musical Group |
---------------------------------------
| Secretary Curch | council |
---------------------------------------
|Wedding Planer | Organizatin Comite|

When accessing Eloquent relationships as properties, the relationship data is "lazy loaded". This means the relationship data is not actually loaded until you first access the property. However, Eloquent can "eager load" relationships at the time you query the parent model. Eager loading alleviates the N + 1 query problem. To illustrate the N + 1 query problem, consider a Chretien model that is related to Poste:
<?php
namespace App;
use Illuminate\Database\Eloquent\Model;
class Poste extends Model
{
/**
* Get the chretien that wrote the poste.
*/
public function chretien()
{
return $this->belongsTo('App\Chretien');
}
}
Now, let's retrieve all chretiens and their postes:
$chretiens = App\Chretien::with('postes')->get();
foreach ($chretiens as $chretien) {
echo $chretien->postes->name;
}
For this operation, only two queries will be executed:
select * from chretiens
select * from postes where id in (1, 2, 3, 4, 5, ...)
Nested Eager Loading
To eager load nested relationships, you may use "dot" syntax. For example, let's eager load all of the poste's and all of the departament's in one Eloquent statement:
$chretiens = App\Chretien::with('postes.departaments')->get();

Related

Why are all my SQL queries being duplicated 4 times for Django using "Prefetch_related" for nested MPTT children?

I have a Child MPTT model that has a ForeignKey to itself:
class Child(MPTTModel):
title = models.CharField(max_length=255)
parent = TreeForeignKey(
"self", on_delete=models.CASCADE, null=True, blank=True, related_name="children"
)
I have a recursive Serializer as I want to show all levels of children for any given Child:
class ChildrenSerializer(serializers.HyperlinkedModelSerializer):
url = HyperlinkedIdentityField(
view_name="app:children-detail", lookup_field="pk"
)
class Meta:
model = Child
fields = ("url", "title", "children")
def get_fields(self):
fields = super(ChildrenSerializer, self).get_fields()
fields["children"] = ChildrenSerializer(many=True)
return fields
I am trying to reduce the number of duplicate/similar queries made when accessing a Child's DetailView.
The view below works for a depth of 2 - however, the "depth" is not always known or static.
class ChildrenDetailView(generics.RetrieveUpdateDestroyAPIView):
queryset = Child.objects.prefetch_related(
"children",
"children__children",
# A depth of 3 will additionally require "children__children__children",
# A depth of 4 will additionally require "children__children__children__children",
# etc.
)
serializer_class = ChildrenSerializer
lookup_field = "pk"
Note: If I don't use prefetch_related and simply set the queryset as Child.objects.all(), every SQL query is duplicated four times... which I have no idea why.
How do I leverage a Child's depth (i.e. the Child's MPTT level field) to optimize prefetching? Should I be overwriting the view's get_object and/or retrieve?
Does it even matter if I add a ridiculous number of depths to the prefetch? E.g. children__children__children__children__children__children__children__children? It doesn't seem to increase the number of queries for Children objects that don't require that level of depth.
Edit:
Hm, not sure why but when I try to serialize any Child's top parent (i.e. MPTT's get_root), it duplicates the SQL query four times???
class Child(MPTTModel):
...
#property
def top_parent(self):
return self.get_root()
class ChildrenSerializer(serializers.HyperlinkedModelSerializer):
...
top_parent = ParentSerializer()
fields = ("url", "title", "children", "top_parent")
Edit 2
Adding an arbitrary SerializerMethodField confirms it's being queried four times... for some reason? e.g.
class ChildrenSerializer(serializers.HyperlinkedModelSerializer):
...
foo = serializers.SerializerMethodField()
def get_foo(self, obj):
print("bar")
return obj.get_root().title
This will print "bar" four times. The SQL query is also repeated four times according to django-debug-toolbar:
SELECT ••• FROM "app_child" WHERE ("app_child"."parent_id" IS NULL AND "app_child"."tree_id" = '7') LIMIT 21
4 similar queries. Duplicated 4 times.
Are you using DRF's browsable API? It initializes serializer 3 more times for HTML forms, in rest_framework.renderers.BrowsableAPIRenderer.get_context.
If you do the same request with, say, Postman, "bar" should get printed only once.

Join multiple tables used as Indexing - Laravel

I have three tables - Chairman, Designation, Members.
MY requirement is to map the member to chairman and assign member a role.
I was able to fetch the list of members under the chairman when I had chairman_id and designation_id in the members table.
Since the chairman change, most of the members stay intact. So I came up with an idea of indexing them
Table - membermap
id | chairman_id | designation_id | member_id
So the list is preserved how many chairmans come and go. I dont need to create new profile for new chairman rather than map to it.
I am now sure how do I do it,
So far I was able to pull the ID but I am not sure how do I join the tables
Tables
Chairman
id| name
Designation
id|designation
Members
id|members
Here is my controller
$mapmember = Statechairman::findOrFail($id)->statechairmembersmap;
dd($mapmember);
In this Iam getting the statechairmembersmap but it's fetching all the result and not limiting the match.
I also tried to join the query using the DB
$mapmember = DB::table('statechairmen')
->join('state_chairman_members_maps', 'state_chairman_members_maps.chairman_id','statechairmen.id')
->join('statemembers','statemembers.id','state_chairman_members_maps.members_id')
->select('state_chairman_members_maps.*')->get();
but this result show me the Table - membermap but not the other results.
My Models:
Chairman :
public function statechairmembersmap(){
return $this->hasMany('App\StateChairmanMembersMap','chairman_id','id');
}
public function statemembers(){
return $this->hasMany('App\Statemembers','chairman_id', 'id');
}
public function statedesignation(){
return $this->hasMany('App\Statedesignation','id','designation_id');
}
membermap:
protected $table = 'state_chairman_members_maps';
protected $dates = ['deleted_at'];
public function statechairman(){
return $this->belongsTo('App\Statechairman','id');
}
public function statedesignations(){
return $this->belongsTo('App\Statedesignation','designation_id','id');
}
public function statemembers(){
return $this->belongsTo('App\Statemembers','members_id','id');
}
Please assist me where I doing wrong.
Thanks a lot for checking the question out.
Finally after a lot of strugle, I was able to find it by myself.
$mapmembers = DB::table('state_chairman_members_maps')
->join('statechairmen','statechairmen.id','=','state_chairman_members_maps.chairman_id')
->join('statemembers','statemembers.id','=','state_chairman_members_maps.members_id')
->join('statedesignations','statedesignations.id','=','state_chairman_members_maps.designation_id')
->where('chairman_id','=',$id)
->get();
Here is what I came up with.
Here I have joined 3 tables and mapped the id comming from the chairman to filter the result. I getting the results.

symfony (doctrine): DQL query, how to exclude results?

I have very basic knowledge of SQL and don't how to modify my query so it returns the expected result.
My entities Activity has a field members (a manyToMany relation). An activity can be done by several members.
I want to select all the activities a specific member does NOT take part into. I know the member's id (my variable $selfId).
Here's what I'm doing in ActivityRepository:
public function getCollectiveActivities($selfId)
{
$qb = $this->createQueryBuilder('a');
$qb->join('a.members', 'm')
->where('m.id != :selfId')
->setParameter('selfId', $selfId);
return $qb
->getQuery()
->getResult();
}
My problem: when an activity "owns" two or more members, if one of them has $selfId as id, this activity ends up in my query results.
But since one of the members of this activity has the id $selfId, it should not be returned. What am I doing wrong?
EDIT
I think I need to be more restrictive, I just don't know how.
Let's say I have two activities:
activity1 which is owned by member1 and member2
activity2 which is owned by member3 and member4
$selfId = 3 (this means I don't want to fetch activities owned by member3)
From what I understand, the join clause might return lines like these:
activity1 | memberId: 1 (different from selfId = 3 so activity1 will be fetched)
activity1 | memberId: 2 (different from selfId = 3 so activity1 will be fetched)
activity2 | memberId: 3 (= selfId so activity2 shouldn't be fetched)
activity2 | memberId: 4 (different from selfId = 3 so activity2 WILL be fetched. PROBLEM???)
EDIT 2
Others already faced the same problem and found this solution and this one, but they seem a bit hacky. Any clue on how to improve them would be welcome.
You have to specifically select the results you want to be hydrated. The problem you're seeing is that you're just selecting activity.
Then when you call $activity->getMembers() members are lazy loaded, and this doesn't take into account your query.
You can avoid this like so:
public function getCollectiveActivities($selfId)
{
$qb = $this->createQueryBuilder('a');
$qb->addSelect('m');
$qb->join('a.members', 'm')
->where('m.id != :selfId')
->setParameter('selfId', $selfId);
return $qb
->getQuery()
->getResult();
}
This means your fetched activity will already have its members hydrated and restricted by your query condition.
I ended up adapting a solution posted by #NtskX.
public function getCollectiveActivities($selfId)
{
$qbMyCollectives = $this->createQueryBuilder('ca');
$qbMyCollectives
->select('ca.id')
->leftJoin('ca.members', 'm')
->where('m.id = :selfId')
->setParameter('selfId', $selfId);
$qb = $this->createQueryBuilder('a');
$qb
->where($qb->expr()->notIn('a.id', $qbMyCollectives->getDQL()))
->setParameter('selfId', $selfId); // I had to declare the parameter one more time
return $qb
->getQuery()
->getResult();
}
I really thought someone would show up with a better way to do the join, so any other answer is much welcome.
public function getCollectiveActivities($selfId)
{
return $this->createQueryBuilder('a')
->join('a.members', 'm')
->where($qb->expr()->neq('c.selfId', $selfId))
->getQuery()
->getResult();
}

Generating seed code from existing database in ASP.NET MVC

I wondered if anyone has encountered a similar challenge:
I have a database with some data that was ETL'ed (imported and transformed) in there from an Excel file. In my ASP.NET MVC web application I'm using Code First approach and dropping/creating every time database changes:
#if DEBUG
Database.SetInitializer(new DropCreateDatabaseIfModelChanges<MyDataContext>());
#endif
However, since the data in the Database is lost, I have to ETL it again, which is annoying.
Since, the DB will be dropped only on model change, I will have to tweak my ETL anyway, I know that. But I'd rather change my DB seed code.
Does anyone know how to take the contents of the database and generate seed code, assuming that both Models and SQL Tables are up to date?
EDIT 1:
I'm planning to use the auto-generated Configuration.cs, and its Seed method, and then use AddOrUpdate() method to add data into the database: Here is Microsoft's Tutorial on migrations (specifically the "Set up the Seed method" section).
Lets say we have a simple database table with 3750 records in it;
| Id | Age | FullName |
|------|-----|-----------------|
| 1 | 50 | Michael Jackson |
| 2 | 42 | Elvis Presley |
| 3 | 48 | Whitney Houston |
| ... | ... | ... |
| 3750 | 57 | Prince |
We want to create this table in our database with using auto-generated Configuration.cs file and its Seed() method.
protected override void Seed(OurDbContainer context)
{
context.GreatestSingers.AddOrUpdate(
p => p.Id,
new GreatestSinger { Id = 1, Age = 50, FullName = "Michael Jackson" },
new GreatestSinger { Id = 2, Age = 42, FullName = "Elvis Presley" },
new GreatestSinger { Id = 3, Age = 48, FullName = "Whitney Houston" }
);
}
This is what you should do. 3750 times!
But you already have this data in your existing database table. So we can use this existing data to create Seed() codes.
With the help of SQL String Concatenation;
SELECT
CONCAT('new GreatestSinger { Id = ', Id ,', Age = ', Age ,', FullName = "', FullName ,'" },')
FROM GreatestSinger
will give us all the code needed to create 3750 rows of data.
Just copy/paste it into Seed() method. And from Package Manager Console;
Add-Migration SeedDBwithSingersData
Update-Database
Another way of seeding data is to run it as sql in an Up migration.
I have code that will read a sql file and run it
using System;
using System.Data.Entity.Migrations;
using System.IO;
public partial class InsertStandingData : DbMigration
{
public override void Up()
{
var baseDir = AppDomain.CurrentDomain
.BaseDirectory
.Replace("\\bin", string.Empty) + "\\Data\\Sql Scripts";
Sql(File.ReadAllText(baseDir + "\\StandingData.sql"));
}
public override void Down()
{
//Add delete sql here
}
}
So if your ETL generates sql for you then you could use that technique.
The advantages of doing it in the Up method are
It will be quicker than doing it using AddOrUpdate because
AddOrUpdate queries the database each time it is called to get any
already existing entity.
You are normally going from a known state (e.g. empty tables) so you probably
don't need to check whether data exists already. NB to ensure this
then you should delete the data in the Down method so that you can
tear all the way down and back up again.
The Up method does not run every time the application starts.
The Seed method provides convenience - and it has the advantage (!?) that it runs every time the application starts
But if you prefer to run the sql from there use ExecuteSqlCommand instead of Sql:
string baseDir = AppDomain.CurrentDomain.BaseDirectory.Replace("\\bin", string.Empty)
+ "\\Data\\Sql Scripts";
string path = Path.Combine(baseDir, "StandingData");
foreach (string file in Directory.GetFiles(path, "*.sql"))
{
context.Database.ExecuteSqlCommand(File.ReadAllText(file));
}
References:
Best way to incrementally seed data
Preparing for database deployment
Database Initializer and Migrations Seed Methods

Simple SQL to Eloquent Query (Laravel)

I have two tables: users (Users) and groups (Groups).
Users
-----------------
id | username | group
1 | Bob | 2
Groups
-----------------
id | name
1 | firstgroup
2 | secondgroup
I would like to display: users.ID, users.username, group.name (1, Bob, secondgroup)
An SQL statement like so would work:
SELECT Users.id, Users.username, Groups.name
FROM Users
INNER JOIN
Groups ON Groups.id = Users.group
However, I'm struggling to write this in Eloquent, since there is no "FROM". At the moment I'm going for something along the lines of the below, using JOINS (http://laravel.com/docs/queries#joins)
$users = Users::select('id','username', 'Groups.name')->joins('Groups.id', '=', 'id')->get();
Now this isn't working - I think the joins has to come before the select but I just can't work it out :(
I think you're confusing a few things here...
You're mixing Eloquent with the lower-level DB::table('foo')->select() syntax. When you want to use Eloquent I suggest you take a look at the docs about relationships in Eloquent.
You should define your models like so:
class User extends Eloquent {
public function group()
{
return $this->belongsTo('Group', 'group');
// second parameter is necessary because you didnt
// name the column "group_id" but simply "group"
}
}
class Group extends Eloquent {
public function users()
{
return $this->hasMany('User', 'group');
}
}
This sets up all the joins you might be needing later. You can then simply use User::with('group')->all(); and have the query built and run for you.
Database: Query Builder(DB) is not a Eloquent(ORM):
Database query builder you have to inform the table names and the fields, like it says on in your related link of laravel docs: "...provides a convenient, fluent interface to creating and running database queries." like these query below:
$users = DB::table('users')
->join('contacts', 'users.id', '=', 'contacts.user_id')
->join('orders', 'users.id', '=', 'orders.user_id')
->select('users.*', 'contacts.phone', 'orders.price')
->get();
Eloquent is a ORM - Object related Mapping, it means that your class User is related to the table users (look at you files Migrations) and this class extends the Model Class, thus you can access the methods like these bellow:
class User extends Models
{
public static function usersWithGroups(){
return User::select('id', 'name', 'email')->with('groups')->get();
}
}
Observe that method is into the class User, so you can access that in a static way "User::", using Eloquent you'll have many hidden static methods that will improve you time codding, because you are inheriting de Model methods, to more details visit the Eloquent Docs at: Eloquent Docs