HQL query and SELECT N+1 issue when using IN - sql

I have some "SELECT n+1" issues with an HSQL query using Hibernate and Spring Data using the IN clause.
We have some organizations that published applications thought a PublishedApplication object.
The current model is (and it cannot be changed easily):
public class Application {
Organization organization;
...
}
public class Organization {
...
}
public class PublishedApplication extends ... {
#OneToOne(cascade = CascadeType.ALL, orphanRemoval = true)
private Application application;
#ManyToOne(cascade = CascadeType.REFRESH)
#JoinColumn(name="publishingorganization_uuid")
protected Organization publishingOrganization;
}
I want to list all apps that are published by one of my organizations.
So I'v write this query:
#Query(value = "SELECT DISTINCT p.application
FROM PublishedApplication p
WHERE p.publishingOrganization IN ?1")
List<Application> findAllByPublishedOrganizationIn(List<Organization> organizations);
This works well but when I enable SQL log, I see:
1 SELECT to get "publishedapplication.application": ok it's the main request.
3 SELECT to get "organization": I have access to 10 organizations but only 3 have published an application
Here is the code call:
log.debug("Before");
List<Application> applications = applicationRepository.findAllByPublishedOrganizationIn(organizations);
log.debug("After");
And here is the Log:
ApplicationService : Before
Hibernate: select distinct applicatio1_.uuid as uuid1_0_, applicatio1_.version as version2_0_, applicatio1_.description as descript3_0_, applicatio1_.lastversiondate as lastvers4_0_, applicatio1_.name as name5_0_, applicatio1_.organization_uuid as organiza6_0_ from publishedapplication publisheda0_ inner join applications applicatio1_ on publisheda0_.application_uuid=applicatio1_.uuid where publisheda0_.publishingorganization_uuid in (? , ? , ? , ? , ? , ? , ? , ? , ? , ?)
Hibernate: select organizati0_.uuid as uuid1_8_0_, organizati0_.version as version2_8_0_, organizati0_.description as descript3_8_0_, organizati0_.name as name4_8_0_ from organizations organizati0_ where organizati0_.uuid=?
Hibernate: select organizati0_.uuid as uuid1_8_0_, organizati0_.version as version2_8_0_, organizati0_.description as descript3_8_0_, organizati0_.name as name4_8_0_ from organizations organizati0_ where organizati0_.uuid=?
Hibernate: select organizati0_.uuid as uuid1_8_0_, organizati0_.version as version2_8_0_, organizati0_.description as descript3_8_0_, organizati0_.name as name4_8_0_ from organizations organizati0_ where organizati0_.uuid=?
ApplicationService : After
I suppose that the 3 'SELECT organization' are triggered by doing this "p.publishingOrganization IN".
Is it a way to resolve this problem using a single SELECT?
Something like this in SQL:
List<Organization> orgs = ...
List<String> orgsIds = ...
doSQL("SELECT *
FROM publishedApplication p
WHERE p.publishingOrganizationId IN (" + orgsIds.join(",") + ")");
Thanks
EDIT: I resolve this using:
SELECT DISTINCT p.application FROM PublishedApplication p JOIN FETCH p.application.organization WHERE p.publishingOrganization IN ?1

Related

Spring Data JPA Query for inner join table throwing error

Spring DATA JPA question... I am trying to write a query to access the data in my sql join table.
I have my join table set up as follows:
#Entity
#Table(name="WritingCompany")
public class WritingCompany {
#Id
#GeneratedValue(strategy = GenerationType.IDENTITY)
#Column(name = "companyId")
private Long id;
// private String companyName;
#ManyToMany
#JoinTable(name = "Join-WritingCo-Carrier",
joinColumns = #JoinColumn(name="writingCo"),
inverseJoinColumns = #JoinColumn(name="carrier")
)
private Set<CarrierAppointment> carriers;
//...getters and setters
}
public class CarrierAppointment {
// ...
#ManyToMany(
cascade=CascadeType.ALL,
mappedBy = "companyName"
)
private Set<WritingCompany> companies;
public Set<WritingCompany> getCompanies() {
return companies;
}
public void setCompanies(Set<WritingCompany> companies) {
this.companies = companies;
}
//...
}
I am unsure which class repository I need to write the query for... I am trying to find all the writingCo names from the join table that all have the same carrier id that matches one specific carrier.
The Join Table is set up through writing company but I feel like it should be accessed through CarrierAppointment Repository since I am matching carrier Id's.
This is what I've tried in the CarrierAppointmentrepository and it throws this error (unexpected token: Join near line 1, column 94):
#Query("Select companyName FROM WritingCompany INNER JOIN CarrierAppointment ON Join-WritingCo-Carrier.carrier= CarrierAppointment.CarrierAppointmentId")
List<CarrierAppointment> findCarrierAppoinmentFromJoin(CarrierAppointment carrier);
I've also tried:
#Query("SELECT writingCo FROM Join-WritingCo-Carrier WHERE carrier= CarrierAppointment.CarrierAppointmentId")
List<CarrierAppointment> findCarrierAppoinmentFromJoin(CarrierAppointment carrier);
Then I tried this in the writingCompanyRepository which also throws a similar error:
#Query("Select companyName FROM WritingCompany INNER JOIN CarrierAppointment ON Join-WritingCo-Carrier.carrier= CarrierAppointment.CarrierAppointmentId")
List<WritingCompany> findAllWithDescriptionQuery(CarrierAppointment carrier);
I am having a hard time understanding what this query is saying. Do I ever need to access the columns from the sql join table, or am I just querying around the join table by asking for each class that is making up the join columns in the join table? What is the right part of the statement, after INNER JOIN stating ? Could someone please provide a deeper explanation of why the query is written so I can figure out why it's not working? I've been reading a lot of inner join examples and just can't seem to figure it out.

Moodle get SQL data but don't get all

I'm working on a plugin for showing all users completed courses.
But I only get 10 records, when I place the SQL inside my database I get 40+. I think there is a limit or it does only return 1 course from every user.
Any tips?
externallib.php file:
/**
* External Web Service Template
*
* #package specialist_in_websites
* #copyright 2011 Moodle Pty Ltd (http://moodle.com)
* #license http://www.gnu.org/copyleft/gpl.html GNU GPL v3 or later
*/
require_once($CFG->libdir . "/externallib.php");
class specialist_in_websites_users_courses_external extends external_api
{
/**
* Returns description of method parameters
* #return external_function_parameters
*/
public static function hello_world_parameters()
{
return new external_function_parameters(
array()
);
}
/**
* Returns welcome message
* #return string welcome message
*/
public static function hello_world()
{
global $DB;
$sql = 'SELECT u.id as user_id, c.id as course_id, DATE_FORMAT(FROM_UNIXTIME(p.timecompleted),"%Y-%m-%d") AS completed_time
FROM mdl_course_completions AS p
JOIN mdl_course AS c ON p.course = c.id
JOIN mdl_user AS u ON p.userid = u.id
WHERE c.enablecompletion = 1 ORDER BY u.id';
$datas = $DB->get_records_sql($sql);
return (array)$datas;
}
/**
* Returns description of method result value
* #return external_description
*/
public static function hello_world_returns()
{
return new external_multiple_structure(
new external_single_structure(
array(
'user_id' => new external_value(PARAM_INT, 'user id'),
'course_id' => new external_value(PARAM_INT, 'course id'),
'completed_time' => new external_value(PARAM_TEXT, 'Time of Completed'),
)
)
);
}
}
and response code:
string(510) "[{"user_id":2,"course_id":12,"completed_time":null},{"user_id":3,"course_id":10,"completed_time":null},{"user_id":4,"course_id":9,"completed_time":null},{"user_id":5,"course_id":41,"completed_time":null},{"user_id":6,"course_id":14,"completed_time":null},{"user_id":7,"course_id":10,"completed_time":null},{"user_id":8,"course_id":9,"completed_time":null},{"user_id":9,"course_id":9,"completed_time":null},{"user_id":10,"course_id":10,"completed_time":null},{"user_id":11,"course_id":10,"completed_time":null}]"
As stated in the docs the $DB->get_records_*() functions get a hashed array of records, indexed by the first field returned.
So, if you return more than one record with the same user_id field, then they will overwrite the previous record in the array.
Make sure you turn on debugging when developing for Moodle and you will see warning messages about this.
Solutions - either find a different field to be the first field returned (e.g. course_completions.id) or use $DB->get_recordset_sql() and then loop through the results.
Also, do not use the 'mdl_' prefix in your code, as that will break on any site with a different prefix, or if you try to run any behat or unit tests on your site. As stated in the docs, write:
FROM {course_completions} p
JOIN {course} c ON p.course = c.id
etc.
(and don't use AS with tables, as that's not compatible with all DB types)

How to Join with Native SQL Query and doctrine

I'm developping an application with symfony 3.4. I want to execute a specific query. So i have two entities: the first is PrPurchaseRequest. the second is PrSpecificFieldValue. PrPurchaseRequest has oneToMany prSpecificFieldValues.
I want to get id of purchaseRequest and prSpecificFieldValues
i did that
$queryBuilder = $this->getEntityManager()->createQuery('select p.id as purchaseId, pr.keyField AS keyField,pr.ID AS prkeyvalueid from '.PrPurchaseRequest::class. ' p LEFT JOIN '. PrSpecificFieldValue::class .' spec ON p.id = spec.purchaseId ');
and that didn't work for me
[Syntax Error] Error: Expected end of string, got
'ON'
how can i do it
Using doctrine you need to play around your entities and their mappings with other entities in order to relate them like
use Doctrine\Common\Collections\ArrayCollection;
/** #Entity */
class PrPurchaseRequest
{
/**
*
* #OneToMany(targetEntity="PrSpecificFieldValue", mappedBy="prPurchaseRequest")
*/
private $prSpecificFieldValues;
// ...
public function __construct() {
$this->prSpecificFieldValues = new ArrayCollection();
}
}
/** #Entity */
class PrSpecificFieldValue
{
/**
*
* #ManyToOne(targetEntity="PrPurchaseRequest", inversedBy="prSpecificFieldValues")
* #JoinColumn(name="pr_purchase_request_id", referencedColumnName="id")
*/
private $prPurchaseRequest;
}
Now you have defined relationship between your entities you can join them based on their mapping (prSpecificFieldValues defined on PrPurchaseRequest class ) like
Its DQL (DQL != SQL)
SELECT p,v
FROM PrPurchaseRequest p
JOIN p.prSpecificFieldValues v
No need to specify ON clause doctrine will handle this for you.
One-To-Many, Bidirectional

How to map ONE-TO-MANY native query result into a POJO class using #SqlResultSetMapping

Im working in a backend API using Java and MySql, and I'm trying to use #SqlResultSetMapping in JPA 2.1 for mapping a ONE-TO-MANY native query result into a POJO class, this is the native query:
#NamedNativeQuery(name = "User.getAll”, query = "SELECT DISTINCT t1.ID, t1.RELIGION_ID t1.gender,t1.NAME,t1.CITY_ID , t2.question_id, t2.answer_id FROM user_table t1 inner join user_answer_table t2 on t1.ID = t2.User_ID“,resultSetMapping="userMapping")
And, here is my result SQL mapping:
#SqlResultSetMapping(
name = "userMapping",
classes = {
#ConstructorResult(
targetClass = MiniUser.class,
columns = {
#ColumnResult(name = "id"),
#ColumnResult(name = "religion_id"),
#ColumnResult(name = "gender"),
#ColumnResult(name = "answers"),
#ColumnResult(name = "name"),
#ColumnResult(name = "city_id")
}
),
#ConstructorResult(
targetClass = MiniUserAnswer.class,
columns = {
#ColumnResult(name = "question_id"),
#ColumnResult(name = "answer_id")
}
)
})
And, here is the implementation of the POJO classes: (I just removed the constructor and the getters/setter)
MiniUser class
public class MiniUser {
String id;
String religionId;
Gender gender;
List<MiniUserAnswer> answers;
String name;
String city_id;
}
and the MiniUserAnswer class
public class MiniUserAnswer {
String questionId;
String answerId;
}
My goal is to execute this Query and return a list of MiniUser, and in each MiniUser: a list of his “answers", which is a list of MiniUserAnswer.
after running this code, I got this error:
The column result [answers] was not found in the results of the query.
I know why, it's because there is no “answers" field in the query select statement.
So, how can I accomplish something like this, considering the performance? This answers list may reach 100.
I really appreciate your help, Thanks in advance!
The query "SELECT DISTINCT t1.ID, t1.RELIGION_ID t1.gender, t1.NAME, t1.CITY_ID, t2.question_id, t2.answer_id" does not return a parameter called answers.
To obtain the result you are looking for I would use:
Option 1 (Criteria Builder)
CriteriaBuilder cb = entityManager.getCriteriaBuilder();
CriteriaQuery<UserTableEntity> cq = cb.createQuery(UserTableEntity.class);
Root<UserTableEntity> rootUserTable = cq.from(UserTableEntity.class);
Join<UserTableEntity,UserAnswerTableEntity> joinAnswerTable = rootUserTable.join(rootUserTable_.id) // if the relationship is defined as lazy, use "fetch" instead of "join"
//cq.where() NO WHERE CLAUSE
cq.select(rootUserTable)
entityManager.createQuery(cq).getResultList();
Option 2 (Named query, not native)
#NamedQuery(name = "User.getAll”, query = "SELECT t1 FROM UserTableEntityt1 join fetch t1.answers)
Option 3 (Entity subgraph, new in JPA 2.1)
In User Entity class:
#NamedEntityGraphs({
#NamedEntityGraph(name = "graph.User.Answers", attributeNodes = #NamedAttributeNode("answers"))
})
In DAO set hints in the entity manager:
EntityGraph graph = this.em.getEntityGraph("graph.User.Answers");
Map hints = new HashMap();
hints.put("javax.persistence.fetchgraph", graph);

Fluent Nhibernate join on non foreign key property

I can't find this anywhere, but it seems pretty trivial. So, please excuse if this is a duplicate.
I have something like:
public class Doctor : Entity
{
...some other properties here...
public virtual string Email { get; set; }
}
public class Lawyer : Entity
{
...some other properties here...
public virtual string Email { get; set; }
}
I want to return all doctors where there is no email match in the Lawyers table like:
select * from Doctors d
where d.Email not in
(select l.Email from Lawyers l where l.Email is not null)
or using a join:
select d.* from Doctors d
left join Lawyers l on l.Email = d.Email
where l.Email is null
The problem is that the Email is of course not set up as a foreign key. I have no mapped property on the Doctor entity that maps to Lawyer.
What I've tried so far:
ICriteria criteria = Session.CreateCriteria(typeof(Doctor))
.CreateAlias("Lawyers.Email", "LawyerEmail", JoinType.LeftOuterJoin)
.Add(Restrictions.IsNull("LawyerEmail"));
return criteria.List<Doctor>();
But, I get a "cannot resolve property Lawyer of MyPlatform.MyNamespace.Doctor" error. Any ideas how to set up my DoctorMap and adjust the criteria tomfoolery to achieve this?
NHibernate for the loss........Entity Framework for the win....
We can achieve that with a feature called subquery:
// a inner SELECT to return all EMAILs from Lawyer table
var subQuery = DetachedCriteria.For<Lawyer>()
.SetProjection(Projections.Property("Email"));
// the root SELECT to get only these Doctors
var criteria = session.CreateCriteria<Doctor>();
// whos email is not in the sub SELECT
criteria.Add(Subqueries.PropertyNotIn("Email", subQuery));
// get first 10
var result = criteria
.SetMaxResults(10)
.SetFirstResult(0) // paging
.List<Doctor>();