I have a LINQ query that makes string search within a few tables. The query however is painfully slow on big tables. At my first attempt, I was getting a timeout. I was able to improve the performance a little. This is the first version of the code:
public ListResponse<UserDTO> GetUsers(FilterParameters filter)
{
var query = from user in _dbContext.Users
.Include(w => w.UserRoles).ThenInclude(u => u.Role)
join accountHolder in _dbContext.AccountHolders
.Include(c => c.OperationCountry)
.Include(x => x.Accounts)
.ThenInclude(x => x.Currency)
on user.Id equals accountHolder.ObjectId into aHolder
from a in aHolder.DefaultIfEmpty()
select new UserDTO
{
Id = user.Id,
FirstName = user.FirstName,
LastName = user.LastName,
Username = user.UserName,
Email = user.Email,
Roles = Mapper.Map<IList<RoleDTO>>(user.UserRoles.Select(i => i.Role)),
LastActivity = user.LastActivity,
CreatedAt = user.CreatedAt,
EmailConfirmed = user.EmailConfirmed,
AccountBalance = a.Accounts.Where(p => p.CurrencyId == a.OperationCountry.LocalCurrencyId).Single().Balance,
AccountReference = a.Accounts.Where(p => p.CurrencyId == a.OperationCountry.LocalCurrencyId).Single().AccountRef
};
// Apply search term
if (!IsNullOrEmpty(filter.SearchTerm))
query = query.Where(w =>
w.FirstName.Contains(filter.SearchTerm)
w.LastName.Contains(filter.SearchTerm) ||
w.Email.Contains(filter.SearchTerm) ||
w.AccountReference.Contains(filter.SearchTerm));
if (filter.ColumnFilters != null)
{
if (filter.ColumnFilters.ContainsKey("EmailConfirmed"))
{
var valueStr = filter.ColumnFilters["EmailConfirmed"];
if (bool.TryParse(valueStr, out var value))
query = query.Where(x => x.EmailConfirmed == value);
}
}
// Get total item count before pagination
var totalItemCount = query.Count();
// Apply pagination
query = query.ApplySortAndPagination(filter);
var userDtoList = query.ToList();
return new ListResponse<UserDTO>()
{
List = userDtoList,
TotalCount = totalItemCount
};
}
I suspected non-database code in the query (such as Single, and Mapping) was causing a slow query so I made an effort to get rid of them. I am still curious how to get a single Account without calling Single() inside the query. Here's the modified version.
public ListResponse<UserDTO> GetUsers(FilterParameters filter)
{
var query = from user in _dbContext.Users
.Include(w => w.UserRoles)
.ThenInclude(u => u.Role)
.Include(w => w.AccountHolder)
.ThenInclude(c => c.OperationCountry)
.Include(w => w.AccountHolder)
.ThenInclude(c => c.Accounts)
.ThenInclude(x => x.Currency)
select user;
if (!IsNullOrEmpty(filter.SearchTerm))
{
query = query.Where(w =>
w.FirstName.StartsWith(filter.SearchTerm) ||
w.LastName.StartsWith(filter.SearchTerm) ||
w.UserName.StartsWith(filter.SearchTerm) ||
w.AccountHolder.Accounts.Any(x => x.AccountRef.StartsWith(filter.SearchTerm)));
}
// total before pagination
var totalItemCount = query.Count();
// Nothing fancy, just OrderBy(filter.OrderBy).Skip(filter.Page).Take(filter.Length)
query = query.ApplySortAndPagination(filter);
userList = query.ToList() //To deal with "Single" calls below, this returns at most filter.Length records
var userDtoResult = (from user in query
select new UserDTO
{
Id = user.Id,
FirstName = user.FirstName,
LastName = user.LastName,
Username = user.UserName,
Email = user.Email,
Roles = Mapper.Map<IList<RoleDTO>>(user.UserRoles.Select(i => i.Role)),
LastActivity = user.LastActivity,
CreatedAt = user.CreatedAt,
EmailConfirmed = user.EmailConfirmed,
AccountBalance = user.AccountHolder.Accounts.Single(p => p.CurrencyId == user.AccountHolder.OperationCountry.LocalCurrencyId).Balance
AccountReference = user.AccountHolder.Accounts.Single(p => p.CurrencyId == user.AccountHolder.OperationCountry.LocalCurrencyId).AccountRef
}).ToList();
return new ListResponse<UserDTO>()
{
List = userDtoResult,
TotalCount = totalItemCount
};
}
The SQL query generated by this query runs slow too, whereas if I write a join query in SQL, it completes in a few hundred milliseconds. I am suspecting I am suffering from N+1 Query problem, but not sure since EF seems to generate a single query when I trace in the SQL Server Profiler.
This is the query generated by the Entity framework and runs in about 8 seconds when I run on the SSMS:
exec sp_executesql N'SELECT TOP(#__p_4) [w].[Id], [w].[AccessFailedCount], [w].[ConcurrencyStamp], [w].[CreatedAt], [w].[CreatedBy], [w].[DeletedAt], [w].[DeletedBy], [w].[DetailId], [w].[Email], [w].[EmailConfirmed], [w].[EmailConfirmedAt], [w].[FacebookId], [w].[FirstName], [w].[GoogleId], [w].[IsActive], [w].[IsDeleted], [w].[LastActivity], [w].[LastName], [w].[LockoutEnabled], [w].[LockoutEnd], [w].[NormalizedEmail], [w].[NormalizedUserName], [w].[Password], [w].[PasswordHash], [w].[PhoneNumber], [w].[PhoneNumberConfirmed], [w].[RoleId], [w].[SecurityStamp], [w].[TwoFactorEnabled], [w].[UpdatedAt], [w].[UpdatedBy], [w].[UserName], [w].[WorkflowId], [t].[Id], [t].[AccountHolderLevel], [t].[AccountHolderType], [t].[CreatedAt], [t].[CreatedBy], [t].[DeletedAt], [t].[DeletedBy], [t].[IsDeleted], [t].[ObjectId], [t].[OperationCountryId], [t].[UpdatedAt], [t].[UpdatedBy], [t0].[Id], [t0].[ContinentId], [t0].[CountryCode], [t0].[CreatedAt], [t0].[CreatedBy], [t0].[DeletedAt], [t0].[DeletedBy], [t0].[ISOCode2], [t0].[IsActive], [t0].[IsDeleted], [t0].[IsOperational], [t0].[LocalCurrencyId], [t0].[Name], [t0].[PhoneCode], [t0].[PostCodeProvider], [t0].[Regex], [t0].[SmsProvider], [t0].[UpdatedAt], [t0].[UpdatedBy]
FROM [Users] AS [w]
LEFT JOIN (
SELECT [a].[Id], [a].[AccountHolderLevel], [a].[AccountHolderType], [a].[CreatedAt], [a].[CreatedBy], [a].[DeletedAt], [a].[DeletedBy], [a].[IsDeleted], [a].[ObjectId], [a].[OperationCountryId], [a].[UpdatedAt], [a].[UpdatedBy]
FROM [AccountHolders] AS [a]
WHERE [a].[IsDeleted] = 0
) AS [t] ON [w].[Id] = [t].[ObjectId]
LEFT JOIN (
SELECT [c].[Id], [c].[ContinentId], [c].[CountryCode], [c].[CreatedAt], [c].[CreatedBy], [c].[DeletedAt], [c].[DeletedBy], [c].[ISOCode2], [c].[IsActive], [c].[IsDeleted], [c].[IsOperational], [c].[LocalCurrencyId], [c].[Name], [c].[PhoneCode], [c].[PostCodeProvider], [c].[Regex], [c].[SmsProvider], [c].[UpdatedAt], [c].[UpdatedBy]
FROM [Countries] AS [c]
WHERE [c].[IsDeleted] = 0
) AS [t0] ON [t].[OperationCountryId] = [t0].[Id]
WHERE ([w].[IsDeleted] = 0) AND ((((([w].[FirstName] LIKE #__filter_SearchTerm_0 + N''%'' AND (LEFT([w].[FirstName], LEN(#__filter_SearchTerm_0)) = #__filter_SearchTerm_0)) OR (#__filter_SearchTerm_0 = N'''')) OR (([w].[LastName] LIKE #__filter_SearchTerm_1 + N''%'' AND (LEFT([w].[LastName], LEN(#__filter_SearchTerm_1)) = #__filter_SearchTerm_1)) OR (#__filter_SearchTerm_1 = N''''))) OR (([w].[UserName] LIKE #__filter_SearchTerm_2 + N''%'' AND (LEFT([w].[UserName], LEN(#__filter_SearchTerm_2)) = #__filter_SearchTerm_2)) OR (#__filter_SearchTerm_2 = N''''))) OR EXISTS (
SELECT 1
FROM [Accounts] AS [x]
WHERE (([x].[IsDeleted] = 0) AND (([x].[AccountRef] LIKE #__filter_SearchTerm_3 + N''%'' AND (LEFT([x].[AccountRef], LEN(#__filter_SearchTerm_3)) = #__filter_SearchTerm_3)) OR (#__filter_SearchTerm_3 = N''''))) AND ([t].[Id] = [x].[AccountHolderId])))
ORDER BY [w].[LastActivity] DESC, [w].[Id], [t].[Id]',N'#__p_4 int,#__filter_SearchTerm_0 nvarchar(100),#__filter_SearchTerm_1 nvarchar(100),#__filter_SearchTerm_2 nvarchar(256),#__filter_SearchTerm_3 nvarchar(450)',#__p_4=10,#__filter_SearchTerm_0=N'james',#__filter_SearchTerm_1=N'james',#__filter_SearchTerm_2=N'james',#__filter_SearchTerm_3=N'james'
Finally this is my SQL query that returns whatever is necessary in less than 100 ms:
declare #searchTerm varchar(100) = '%james%'
select top 10
u.Id,
u.UserName,
u.FirstName,
u.LastName,
u.LastActivity,
u.CreatedAt,
a.Balance,
a.AccountRef,
ah.AccountHolderLevel,
u.Email,
r.Name
from Users u
join AccountHolders ah on ah.ObjectId = u.Id
join Accounts a on ah.Id = a.AccountHolderId
join UserRoles ur on ur.UserId = u.Id
join Roles r on r.Id = ur.RoleId
where FirstName like #searchTerm or LastName like #searchTerm or u.UserName like #searchTerm or FirstName + ' ' + LastName like #searchTerm or a.AccountRef like #searchTerm
and a.CurrencyId = ah.OperationCountryId
The columns I am searching are all indexed by the way, so that's not a problem. I know that the new EF-Core has many performance improvements. Unfortunately, I cannot update due to sheer number of breaking changes.
I am not sure splitting query into 2 (one for users and one for account) would work well, because there will be joins all over again. If I cannot find a solution using I plan converting my query to a view, but I want to do it as a last resort, since our convention is to use EF as much as possible. And I refuse to believe that EF does not have a solution. This is not actually a complex query at all and I am sure a fairly common use case.
So, what is the best way to optimize this query using EF-Core?
So, what is the best way to optimize this query using EF-Core?
Many things have changed in EF Core query pipeline since 2.1 (3.0, 3.1, 5.0 and now working on 6.0), but some general rules can be used, with the goal of getting rid of the client side query evaluation (which starting with 3.0 is not supported at all, so it's good to start preparing for the switch - support for 2.1 ends August this year).
The first would be to remove all these Include / ThenInclude. If the query is projecting the result in DTO without involving entity instances, then all these are redundant/not needed and removing them will ensure the query gets fully translated to SQL.
var query = _dbContext.Users.AsQueryable();
// Apply filters...
The next is the Roles collection. You must remove Mapper.Map call, otherwise it can't be translated. In general either use AutoMapper mappings and ProjectTo to fully handle the projection, or not use it at all (never put Map method calls inside query expression tree). According to your SQL, it should be something like this
Roles = user.UserRoles.Select(ur => ur.Role)
.Select(r => new RoleDTO { Name = r.Name })
.ToList(),
Actually EF Core will execute this as separate query (a behavior broken by "single query mode" in 3.x, and brought back optionally with 6.0 "split query mode"), so it is is important to have ToList() call at the end, otherwise you'll get N + 1 queries rather than 2.
Finally, the Single() call. It can be avoided by flattening the sub collection using correlated SelectMany, or its query syntax equivalent
from user in query
let ah = user.AccountHolder
from a in ah.Accounts
where a.CurrencyId == ah.OperationCountryId
The let statement is not mandatory, I've added it just for readability. Now you can use the range variables user, ah and a in the final select similar to table aliases in SQL.
Also since your SQL query doesn't really enforce single account match, there is no such enforcement in the LINQ query as well. If it was needed, then the equivalent of the Single can be achieved with SelectMany + Where + `Take(1), e.g.
from user in query
let ah = user.AccountHolder
from a in ah.Accounts
.Where(a => a.CurrencyId == ah.OperationCountryId)
.Take(1)
(a mixture of query and method syntax, but LINQ allows that)
So the final query would be something like this
from user in query
let ah = user.AccountHolder
from a in ah.Accounts
where a.CurrencyId == ah.OperationCountryId
select new //UserDTO
{
Id = user.Id,
FirstName = user.FirstName,
LastName = user.LastName,
Username = user.UserName,
Email = user.Email,
Roles = user.UserRoles.Select(ur => ur.Role)
.Select(r => new RoleDTO { Name = r.Name })
.ToList(),
LastActivity = user.LastActivity,
CreatedAt = user.CreatedAt,
EmailConfirmed = user.EmailConfirmed,
AccountBalance = a.Balance,
AccountReference = a.AccountRef
}
and should translate to very similar to the handcrafted SQL. And hopefully execute faster similar to it.
Im working in a backend API using Java and MySql, and I'm trying to use #SqlResultSetMapping in JPA 2.1 for mapping a ONE-TO-MANY native query result into a POJO class, this is the native query:
#NamedNativeQuery(name = "User.getAll”, query = "SELECT DISTINCT t1.ID, t1.RELIGION_ID t1.gender,t1.NAME,t1.CITY_ID , t2.question_id, t2.answer_id FROM user_table t1 inner join user_answer_table t2 on t1.ID = t2.User_ID“,resultSetMapping="userMapping")
And, here is my result SQL mapping:
#SqlResultSetMapping(
name = "userMapping",
classes = {
#ConstructorResult(
targetClass = MiniUser.class,
columns = {
#ColumnResult(name = "id"),
#ColumnResult(name = "religion_id"),
#ColumnResult(name = "gender"),
#ColumnResult(name = "answers"),
#ColumnResult(name = "name"),
#ColumnResult(name = "city_id")
}
),
#ConstructorResult(
targetClass = MiniUserAnswer.class,
columns = {
#ColumnResult(name = "question_id"),
#ColumnResult(name = "answer_id")
}
)
})
And, here is the implementation of the POJO classes: (I just removed the constructor and the getters/setter)
MiniUser class
public class MiniUser {
String id;
String religionId;
Gender gender;
List<MiniUserAnswer> answers;
String name;
String city_id;
}
and the MiniUserAnswer class
public class MiniUserAnswer {
String questionId;
String answerId;
}
My goal is to execute this Query and return a list of MiniUser, and in each MiniUser: a list of his “answers", which is a list of MiniUserAnswer.
after running this code, I got this error:
The column result [answers] was not found in the results of the query.
I know why, it's because there is no “answers" field in the query select statement.
So, how can I accomplish something like this, considering the performance? This answers list may reach 100.
I really appreciate your help, Thanks in advance!
The query "SELECT DISTINCT t1.ID, t1.RELIGION_ID t1.gender, t1.NAME, t1.CITY_ID, t2.question_id, t2.answer_id" does not return a parameter called answers.
To obtain the result you are looking for I would use:
Option 1 (Criteria Builder)
CriteriaBuilder cb = entityManager.getCriteriaBuilder();
CriteriaQuery<UserTableEntity> cq = cb.createQuery(UserTableEntity.class);
Root<UserTableEntity> rootUserTable = cq.from(UserTableEntity.class);
Join<UserTableEntity,UserAnswerTableEntity> joinAnswerTable = rootUserTable.join(rootUserTable_.id) // if the relationship is defined as lazy, use "fetch" instead of "join"
//cq.where() NO WHERE CLAUSE
cq.select(rootUserTable)
entityManager.createQuery(cq).getResultList();
Option 2 (Named query, not native)
#NamedQuery(name = "User.getAll”, query = "SELECT t1 FROM UserTableEntityt1 join fetch t1.answers)
Option 3 (Entity subgraph, new in JPA 2.1)
In User Entity class:
#NamedEntityGraphs({
#NamedEntityGraph(name = "graph.User.Answers", attributeNodes = #NamedAttributeNode("answers"))
})
In DAO set hints in the entity manager:
EntityGraph graph = this.em.getEntityGraph("graph.User.Answers");
Map hints = new HashMap();
hints.put("javax.persistence.fetchgraph", graph);
I have these tables: USERS, USER_ROLES, ROLES, PERMISSIONS, ROLE_PERMISSIONS
I am implementing the:
AssignedRoles (M): returns the set of roles assigned to a given user;
So, if I was going to write the query for this function it would like something like this:
SELECT role_id FROM USER_ROLES WHERE user_id = 'M'
where M is the given user id, then I would look up each role by their id and return the Role object, or I would use a join, but that is not relevant here.
so where is what my UserRole model looks like:
case class UserRole(id:Long, userID:Long, roleID:Long)
object UserRoles {
class UserRoles(tag: Tag) extends Table[UserRole](tag, "USER_ROLES") {
def id = column[Long]("ID", O.PrimaryKey, O.AutoInc)
def userID = column[Long]("USER_ID")
def roleID = column[Long]("ROLE_ID")
def user = foreignKey("USER_ID", userID, TableQuery[Users])(_.id)
def role = foreignKey("ROLE_ID", roleID, TableQuery[Roles])(_.id)
def * = (id, userID, roleID) <> (UserRole.tupled, UserRole.unapply)
}
def retrieveRoles(userID:Long) : List[Role] = {
DB.withSession { implicit session =>
userRoles.filter(_.userID === userID).list
}
}
As expected retrieveRoles returns a list of UserRoles, but I want a list of Roles, so I woulds have to write another query that will take UserRole.roleID and find it in the roles table for each of UserRole that is returned. That is a lot of queries and I feel like there is a way that will to do this in one query. Any help from the Slick experts?
userRoles.filter(_.userID === userID).flatMap(_.role)
I have a DAO User :
class User{
int id,
int name,
List<Vehicle> vehicles;
}
In my UserMapper.java interface, I have a method that gives me User DAO :
#Select("select u.id, u.name, v.id, v.name from User u, Vehicle v where u.id=#{id} and v.user_id=u.id")
public User getUser(int id);
How to make sure that the above query fills the User object's vehicle collection properly ?
Please use annotations (and XML only as a last resort)
From Mybatis 3 User Guide,
Note: You will notice that join mapping is not supported via the Annotations
API. This is due to the limitation in Java Annotations that does not
allow for circular references.
Best you can do is something like this,
#Results({
#Result(property = "id", column = "id", id = true),
#Result(property = "name", column = "name"),
#Result(many=#Many(select="your select"), javaType=Vehicle.class)
})
#Select("select u.id, u.name, v.id, v.name from User u, Vehicle v where u.id=#{id} and v.user_id=u.id")
public User getUser(int id);
My advice, use the xml.
I'm trying to create a criteria query that grabs "RejectedRecords" by useruploaded that are not flagged as being deleted or the Facility in the RejectedRecord is in a list of facilities that a user is assigned to (user.UserFacilities). I have the first part working fine (By User and Not Deleted) but I'm not sure how to add the OR clause to take records that are in the collection of user-facilities. In SQL it would look like:
SELECT *
FROM RejectedRecords
WHERE (UserUploaded = 1 AND IsDeleted = 0)
OR FacilityId IN (SELECT FacilityId FROM UserFacility WHERE UserId = 1)
Here's my attempt in C# (Not sure how to perform the subquery):
public IList<RejectedRecord> GetRejectedRecordsByUser(User u)
{
return base._session.CreateCriteria(typeof(RejectedRecord))
.Add(
(
Expression.Eq(RejectedRecord.MappingNames.UserUploaded, u)
&& Expression.Eq(RejectedRecord.MappingNames.IsDeleted, false)
)
)
.List<RejectedRecord>();
}
The Key is to use Disjunction and Conjunction combined with a Subquery.
var facilityIdQuery = DetachedCriteria.For<UserFacility>()
.Add(Expression.Eq("User.Id", u))
.SetProjection(Projections.Property("Facility.Id"));
var results = session.CreateCriteria<RejectedRecords>()
.Add(
Restrictions.Disjunction()
.Add(
Restrictions.And(
Restrictions.Eq(RejectedRecord.MappingNames.UserUploaded, u),
Restrictions.Eq(RejectedRecord.MappingNames.IsDeleted, false)
)
)
.Add(Subqueries.PropertyIn("FacilityId",facilityIdQuery))
).List();