I have a LINQ query that makes string search within a few tables. The query however is painfully slow on big tables. At my first attempt, I was getting a timeout. I was able to improve the performance a little. This is the first version of the code:
public ListResponse<UserDTO> GetUsers(FilterParameters filter)
{
var query = from user in _dbContext.Users
.Include(w => w.UserRoles).ThenInclude(u => u.Role)
join accountHolder in _dbContext.AccountHolders
.Include(c => c.OperationCountry)
.Include(x => x.Accounts)
.ThenInclude(x => x.Currency)
on user.Id equals accountHolder.ObjectId into aHolder
from a in aHolder.DefaultIfEmpty()
select new UserDTO
{
Id = user.Id,
FirstName = user.FirstName,
LastName = user.LastName,
Username = user.UserName,
Email = user.Email,
Roles = Mapper.Map<IList<RoleDTO>>(user.UserRoles.Select(i => i.Role)),
LastActivity = user.LastActivity,
CreatedAt = user.CreatedAt,
EmailConfirmed = user.EmailConfirmed,
AccountBalance = a.Accounts.Where(p => p.CurrencyId == a.OperationCountry.LocalCurrencyId).Single().Balance,
AccountReference = a.Accounts.Where(p => p.CurrencyId == a.OperationCountry.LocalCurrencyId).Single().AccountRef
};
// Apply search term
if (!IsNullOrEmpty(filter.SearchTerm))
query = query.Where(w =>
w.FirstName.Contains(filter.SearchTerm)
w.LastName.Contains(filter.SearchTerm) ||
w.Email.Contains(filter.SearchTerm) ||
w.AccountReference.Contains(filter.SearchTerm));
if (filter.ColumnFilters != null)
{
if (filter.ColumnFilters.ContainsKey("EmailConfirmed"))
{
var valueStr = filter.ColumnFilters["EmailConfirmed"];
if (bool.TryParse(valueStr, out var value))
query = query.Where(x => x.EmailConfirmed == value);
}
}
// Get total item count before pagination
var totalItemCount = query.Count();
// Apply pagination
query = query.ApplySortAndPagination(filter);
var userDtoList = query.ToList();
return new ListResponse<UserDTO>()
{
List = userDtoList,
TotalCount = totalItemCount
};
}
I suspected non-database code in the query (such as Single, and Mapping) was causing a slow query so I made an effort to get rid of them. I am still curious how to get a single Account without calling Single() inside the query. Here's the modified version.
public ListResponse<UserDTO> GetUsers(FilterParameters filter)
{
var query = from user in _dbContext.Users
.Include(w => w.UserRoles)
.ThenInclude(u => u.Role)
.Include(w => w.AccountHolder)
.ThenInclude(c => c.OperationCountry)
.Include(w => w.AccountHolder)
.ThenInclude(c => c.Accounts)
.ThenInclude(x => x.Currency)
select user;
if (!IsNullOrEmpty(filter.SearchTerm))
{
query = query.Where(w =>
w.FirstName.StartsWith(filter.SearchTerm) ||
w.LastName.StartsWith(filter.SearchTerm) ||
w.UserName.StartsWith(filter.SearchTerm) ||
w.AccountHolder.Accounts.Any(x => x.AccountRef.StartsWith(filter.SearchTerm)));
}
// total before pagination
var totalItemCount = query.Count();
// Nothing fancy, just OrderBy(filter.OrderBy).Skip(filter.Page).Take(filter.Length)
query = query.ApplySortAndPagination(filter);
userList = query.ToList() //To deal with "Single" calls below, this returns at most filter.Length records
var userDtoResult = (from user in query
select new UserDTO
{
Id = user.Id,
FirstName = user.FirstName,
LastName = user.LastName,
Username = user.UserName,
Email = user.Email,
Roles = Mapper.Map<IList<RoleDTO>>(user.UserRoles.Select(i => i.Role)),
LastActivity = user.LastActivity,
CreatedAt = user.CreatedAt,
EmailConfirmed = user.EmailConfirmed,
AccountBalance = user.AccountHolder.Accounts.Single(p => p.CurrencyId == user.AccountHolder.OperationCountry.LocalCurrencyId).Balance
AccountReference = user.AccountHolder.Accounts.Single(p => p.CurrencyId == user.AccountHolder.OperationCountry.LocalCurrencyId).AccountRef
}).ToList();
return new ListResponse<UserDTO>()
{
List = userDtoResult,
TotalCount = totalItemCount
};
}
The SQL query generated by this query runs slow too, whereas if I write a join query in SQL, it completes in a few hundred milliseconds. I am suspecting I am suffering from N+1 Query problem, but not sure since EF seems to generate a single query when I trace in the SQL Server Profiler.
This is the query generated by the Entity framework and runs in about 8 seconds when I run on the SSMS:
exec sp_executesql N'SELECT TOP(#__p_4) [w].[Id], [w].[AccessFailedCount], [w].[ConcurrencyStamp], [w].[CreatedAt], [w].[CreatedBy], [w].[DeletedAt], [w].[DeletedBy], [w].[DetailId], [w].[Email], [w].[EmailConfirmed], [w].[EmailConfirmedAt], [w].[FacebookId], [w].[FirstName], [w].[GoogleId], [w].[IsActive], [w].[IsDeleted], [w].[LastActivity], [w].[LastName], [w].[LockoutEnabled], [w].[LockoutEnd], [w].[NormalizedEmail], [w].[NormalizedUserName], [w].[Password], [w].[PasswordHash], [w].[PhoneNumber], [w].[PhoneNumberConfirmed], [w].[RoleId], [w].[SecurityStamp], [w].[TwoFactorEnabled], [w].[UpdatedAt], [w].[UpdatedBy], [w].[UserName], [w].[WorkflowId], [t].[Id], [t].[AccountHolderLevel], [t].[AccountHolderType], [t].[CreatedAt], [t].[CreatedBy], [t].[DeletedAt], [t].[DeletedBy], [t].[IsDeleted], [t].[ObjectId], [t].[OperationCountryId], [t].[UpdatedAt], [t].[UpdatedBy], [t0].[Id], [t0].[ContinentId], [t0].[CountryCode], [t0].[CreatedAt], [t0].[CreatedBy], [t0].[DeletedAt], [t0].[DeletedBy], [t0].[ISOCode2], [t0].[IsActive], [t0].[IsDeleted], [t0].[IsOperational], [t0].[LocalCurrencyId], [t0].[Name], [t0].[PhoneCode], [t0].[PostCodeProvider], [t0].[Regex], [t0].[SmsProvider], [t0].[UpdatedAt], [t0].[UpdatedBy]
FROM [Users] AS [w]
LEFT JOIN (
SELECT [a].[Id], [a].[AccountHolderLevel], [a].[AccountHolderType], [a].[CreatedAt], [a].[CreatedBy], [a].[DeletedAt], [a].[DeletedBy], [a].[IsDeleted], [a].[ObjectId], [a].[OperationCountryId], [a].[UpdatedAt], [a].[UpdatedBy]
FROM [AccountHolders] AS [a]
WHERE [a].[IsDeleted] = 0
) AS [t] ON [w].[Id] = [t].[ObjectId]
LEFT JOIN (
SELECT [c].[Id], [c].[ContinentId], [c].[CountryCode], [c].[CreatedAt], [c].[CreatedBy], [c].[DeletedAt], [c].[DeletedBy], [c].[ISOCode2], [c].[IsActive], [c].[IsDeleted], [c].[IsOperational], [c].[LocalCurrencyId], [c].[Name], [c].[PhoneCode], [c].[PostCodeProvider], [c].[Regex], [c].[SmsProvider], [c].[UpdatedAt], [c].[UpdatedBy]
FROM [Countries] AS [c]
WHERE [c].[IsDeleted] = 0
) AS [t0] ON [t].[OperationCountryId] = [t0].[Id]
WHERE ([w].[IsDeleted] = 0) AND ((((([w].[FirstName] LIKE #__filter_SearchTerm_0 + N''%'' AND (LEFT([w].[FirstName], LEN(#__filter_SearchTerm_0)) = #__filter_SearchTerm_0)) OR (#__filter_SearchTerm_0 = N'''')) OR (([w].[LastName] LIKE #__filter_SearchTerm_1 + N''%'' AND (LEFT([w].[LastName], LEN(#__filter_SearchTerm_1)) = #__filter_SearchTerm_1)) OR (#__filter_SearchTerm_1 = N''''))) OR (([w].[UserName] LIKE #__filter_SearchTerm_2 + N''%'' AND (LEFT([w].[UserName], LEN(#__filter_SearchTerm_2)) = #__filter_SearchTerm_2)) OR (#__filter_SearchTerm_2 = N''''))) OR EXISTS (
SELECT 1
FROM [Accounts] AS [x]
WHERE (([x].[IsDeleted] = 0) AND (([x].[AccountRef] LIKE #__filter_SearchTerm_3 + N''%'' AND (LEFT([x].[AccountRef], LEN(#__filter_SearchTerm_3)) = #__filter_SearchTerm_3)) OR (#__filter_SearchTerm_3 = N''''))) AND ([t].[Id] = [x].[AccountHolderId])))
ORDER BY [w].[LastActivity] DESC, [w].[Id], [t].[Id]',N'#__p_4 int,#__filter_SearchTerm_0 nvarchar(100),#__filter_SearchTerm_1 nvarchar(100),#__filter_SearchTerm_2 nvarchar(256),#__filter_SearchTerm_3 nvarchar(450)',#__p_4=10,#__filter_SearchTerm_0=N'james',#__filter_SearchTerm_1=N'james',#__filter_SearchTerm_2=N'james',#__filter_SearchTerm_3=N'james'
Finally this is my SQL query that returns whatever is necessary in less than 100 ms:
declare #searchTerm varchar(100) = '%james%'
select top 10
u.Id,
u.UserName,
u.FirstName,
u.LastName,
u.LastActivity,
u.CreatedAt,
a.Balance,
a.AccountRef,
ah.AccountHolderLevel,
u.Email,
r.Name
from Users u
join AccountHolders ah on ah.ObjectId = u.Id
join Accounts a on ah.Id = a.AccountHolderId
join UserRoles ur on ur.UserId = u.Id
join Roles r on r.Id = ur.RoleId
where FirstName like #searchTerm or LastName like #searchTerm or u.UserName like #searchTerm or FirstName + ' ' + LastName like #searchTerm or a.AccountRef like #searchTerm
and a.CurrencyId = ah.OperationCountryId
The columns I am searching are all indexed by the way, so that's not a problem. I know that the new EF-Core has many performance improvements. Unfortunately, I cannot update due to sheer number of breaking changes.
I am not sure splitting query into 2 (one for users and one for account) would work well, because there will be joins all over again. If I cannot find a solution using I plan converting my query to a view, but I want to do it as a last resort, since our convention is to use EF as much as possible. And I refuse to believe that EF does not have a solution. This is not actually a complex query at all and I am sure a fairly common use case.
So, what is the best way to optimize this query using EF-Core?
So, what is the best way to optimize this query using EF-Core?
Many things have changed in EF Core query pipeline since 2.1 (3.0, 3.1, 5.0 and now working on 6.0), but some general rules can be used, with the goal of getting rid of the client side query evaluation (which starting with 3.0 is not supported at all, so it's good to start preparing for the switch - support for 2.1 ends August this year).
The first would be to remove all these Include / ThenInclude. If the query is projecting the result in DTO without involving entity instances, then all these are redundant/not needed and removing them will ensure the query gets fully translated to SQL.
var query = _dbContext.Users.AsQueryable();
// Apply filters...
The next is the Roles collection. You must remove Mapper.Map call, otherwise it can't be translated. In general either use AutoMapper mappings and ProjectTo to fully handle the projection, or not use it at all (never put Map method calls inside query expression tree). According to your SQL, it should be something like this
Roles = user.UserRoles.Select(ur => ur.Role)
.Select(r => new RoleDTO { Name = r.Name })
.ToList(),
Actually EF Core will execute this as separate query (a behavior broken by "single query mode" in 3.x, and brought back optionally with 6.0 "split query mode"), so it is is important to have ToList() call at the end, otherwise you'll get N + 1 queries rather than 2.
Finally, the Single() call. It can be avoided by flattening the sub collection using correlated SelectMany, or its query syntax equivalent
from user in query
let ah = user.AccountHolder
from a in ah.Accounts
where a.CurrencyId == ah.OperationCountryId
The let statement is not mandatory, I've added it just for readability. Now you can use the range variables user, ah and a in the final select similar to table aliases in SQL.
Also since your SQL query doesn't really enforce single account match, there is no such enforcement in the LINQ query as well. If it was needed, then the equivalent of the Single can be achieved with SelectMany + Where + `Take(1), e.g.
from user in query
let ah = user.AccountHolder
from a in ah.Accounts
.Where(a => a.CurrencyId == ah.OperationCountryId)
.Take(1)
(a mixture of query and method syntax, but LINQ allows that)
So the final query would be something like this
from user in query
let ah = user.AccountHolder
from a in ah.Accounts
where a.CurrencyId == ah.OperationCountryId
select new //UserDTO
{
Id = user.Id,
FirstName = user.FirstName,
LastName = user.LastName,
Username = user.UserName,
Email = user.Email,
Roles = user.UserRoles.Select(ur => ur.Role)
.Select(r => new RoleDTO { Name = r.Name })
.ToList(),
LastActivity = user.LastActivity,
CreatedAt = user.CreatedAt,
EmailConfirmed = user.EmailConfirmed,
AccountBalance = a.Balance,
AccountReference = a.AccountRef
}
and should translate to very similar to the handcrafted SQL. And hopefully execute faster similar to it.
I have a similar case to the following:
Say there's a number of jobs to be done and for each job there's a history of workers where only one worker is active per job. There's three tables: the Job itself, a mapping table JobWorkers which holds the history of workers for a given job (including a datetime "To" which indicates whether still active (null) or when assignment was cancelled (end date)) and Workers which have a first and last name.
I'd like to query a list of all jobs and the first and last name of the currently assigned worker as flat model. This is the code I'm executing:
var jobExample = dbContext.Jobs.Select(j => new
{
j.JobId,
// ...some other columns from jobs table
j.JobWorker.FirstOrDefault(jw => jw.To == null).Worker.FirstName, // first name of currently assigned worker
j.JobWorker.FirstOrDefault(jw => jw.To == null).Worker.LastName // last name of currently assigned worker
}).First();
The following SQL query is generated:
SELECT TOP (1)
[Extent1].[JobId] AS [JobId],
[Extent3].[FirstName] AS [FirstName],
[Extent5].[LastName] AS [LastName]
FROM [tables].[Jobs] AS [Extent1]
OUTER APPLY (SELECT TOP (1)
[Extent2].[WorkerId] AS [WorkerId]
FROM [tables].[JobWorkers] AS [Extent2]
WHERE ([Extent1].[JobId] = [Extent2].[JobId]) AND ([Extent2].[To] IS NULL) ) AS [Limit1]
LEFT OUTER JOIN [tables].[Workers] AS [Extent3] ON [Limit1].[WorkerId] = [Extent3].[WorkerId]
OUTER APPLY (SELECT TOP (1)
[Extent4].[WorkerId] AS [WorkerId]
FROM [tables].[JobWorkers] AS [Extent4]
WHERE ([Extent1].[JobId] = [Extent4].[JobId]) AND ([Extent4].[To] IS NULL) ) AS [Limit2]
LEFT OUTER JOIN [tables].[Workers] AS [Extent5] ON [Limit2].[WorkerId] = [Extent5].[WorkerId]
As one can see there're two outer apply/left outer joins that are identical. I'd like to get rid of one of those to make the query more performant.
Note that the select statement is dynamically generated based on what information the user actually wants to query. But even if this didn't apply I'm not sure how to do this without having a hierarchic structure and then only afterwards flatten it in .NET
Thanks for your help and if I can improve this question in any way please comment.
You've probably seen that there are two types of LINQ methods: the ones that return IQueryable<...>, and the other ones.
Methods of the first group use deferred execution. This means, that the query is made, but not executed yet. Your database is not contacted.
Methods of the second group, like ToList(), FirstOrDefault(), Count(), Any(), will execute the query: they will contact the database, and fetch the data that is needed to calculate the result.
This is the reason, that you should try to postpone any method of the second group to as last as possible. If you do it earlier, and you do something LINQy after it, changes are that you fetch to much data, or, as in your case: that you do execute the same code twice.
The solution is: move your FirstOrDefault to a later moment.
var jobExample = dbContext.Jobs.Select(job => new
{
Id = job.JobId,
... // other job properties
ActiveWorker = job.JobWorkers
.Where(jobWorker => jobWorker.To == null)
.Select(worker => new
{
FirstName = worker.FirstName,
LastName = worker.LastName,
})
.FirstOrDefault(),
})
.FirstOrDefault();
The result is slightly different than yours:
Id = 10;
... // other Job properties
// the current active worker:
ActiveWorker =
{
FirstName = "John",
LastName = "Doe",
}
If you really want an object with Id / FirstName / LastName, add an extra Select before your final FirstOrDefault:
.Select(jobWithActiveWorker => new
{
Id = jobWithActiveWorker.Id,
... // other Job properties
// properties of the current active worker
FirstName = jobWithActiveWorker.FirstName,
LastName = jobWithActiveWorker.LastName,
})
.FirstOrDefault();
Personally I think that you should not mix Job properties with Worker properties, so I think the first solution: "Job with its currently active worker" is neater: the Job properties are separated from the Worker properties. You can see why that is important if you also wanted the Id of the active worker:
.Select(job => new
{
Id = job.JobId,
... // other job properties
ActiveWorker = job.JobWorkers
.Where(jobWorker => jobWorker.To == null)
.Select(jobworker => new
{
Id = jobworker.Id,
FirstName = jobworker.FirstName,
LastName = jobworker.LastName,
})
.FirstOrDefault(),
})
.FirstOrDefault();
Try rewriting your query like this:
var query =
from j in dbContext.Jobs
let ws = j.JobWorker
.Where(jw => jw.To == null)
.Select(jw => jw.Worker)
.Take(1)
from w in ws.DefaultIfEmpty()
select new
{
j.JobId,
// other properties
w.FirstName,
w.LastName,
};
The query processor probably could not have optimized any further to know it could use the subquery once.
I have the following criteria and detachedCriteria.
var criteria = Session.CreateCriteria<ItemAnalysis>("ia");
criteria.CreateAlias("ia.ItemInstance", "ii");
criteria.CreateAlias("ii.ScoreAdministration", "sa");
criteria.Add(Restrictions.Eq("ii.ItemId", itemId));
var status = DetachedCriteria.For<ItemAnalysis>("ia_")
.CreateAlias("ia_.ItemInstance", "ii_")
.CreateAlias("ii_.ScoreAdministration", "sa_")
.Add(Restrictions.Eq("ii_.ItemId", itemId))
.SetProjection(
Projections.SqlProjection
(
"ia_.CTTItemStatId, RANK() OVER(Partition BY
sa_.ExamSeriesCode ORDER BY ia_.StatDate DESC) AS RowNm",
new string[] { "CTTItemStatId", "RowNm" },
new IType[] { NHibernate.NHibernateUtil.Int32,
NHibernate.NHibernateUtil.Int32 }
)
);
I need a way to get a projection or second subquery from the detached query that has just that property and adds a restriction of RowNm=1. I looked at DetachedCriteria.CreateCriteria but that requires an Association Path. I haven't found any similar examples. I did try
// what parameters should I be using here??
var subQuery = status.CreateCriteria(?, ?) // expects an association path and an alias
.Add(Restrictions.Eq("RowNm", 1))
.SetProjection(
Projections.Property("CTTItemStatId")
);
criteria = criteria.Add(Subqueries.PropertyEq("CTTItemStatId", subQuery));
We are trying to upgrade to NHibernate 3.0 and now i am having problem with the following Linq query. It returns "Only one expression can be specified in the select list when the subquery is not introduced with EXISTS." error.
This is the linq query in the controller.
var list = (from item in ItemTasks.FindTabbedOrDefault(tab)
select new ItemSummary
{
Id = item.Id,
LastModifyDate = item.LastModifyDate,
Tags = (from tag in item.Tags
select new TagSummary
{
ItemsCount = tag.Items.Count,
Name = tag.Name
}).ToList(),
Title = item.Title
});
and the following is the sql generated for this query
select TOP ( 1 /* #p0 */ ) item0_.Id as col_0_0_,
item0_.LastModifyDate as col_1_0_,
(select (select cast(count(* ) as INT)
from dbo.ItemsToTags items3_,
dbo.Item item4_
where tag2_.Id = items3_.Tag_id
and items3_.Item_id = item4_.Id),
tag2_.Name
from dbo.ItemsToTags tags1_,
dbo.Tag tag2_
where item0_.Id = tags1_.Item_id
and tags1_.Tag_id = tag2_.Id) as col_2_0_,
item0_.Title as col_3_0_ from dbo.Item item0_ order by item0_.ItemPostDate desc
ps:If i remove the Tags property in the linq query, it works fine.
Where is the problem in the query?
Thanks in advance.
I've got the same Generic ADO Exception error, I think it's actually the limitation of SQL server;
Is it possible somehow load object graph with projections in collections?
If I try this one:
var cats = q.Select(t => new cat()
{
NickName = t.NickName,
Legs = t.Legs.Select(l => new Leg()
{
Color = l.Color,
Size = l.Size
}).ToList()
}).ToList();
That does the same error..
I'm trying to code the following HQL query using the Criteria API:
var userList = _session
.CreateQuery("select u from User u where u.Role.ID=3 and u.Customer.ID=:cID")
.SetInt32("cID", 1)
.List<User>();
(3 NHibernate objects : User(ID, Name, Role, Customer), Role(ID, Name) and Customer(ID, Name).
I tried the following but it doesn't work because NHibernate tries to find a Customer associated with a Role:
var userList = _session
.CreateCriteria(typeof(User))
.CreateCriteria("Role")
.Add(Restrictions.Eq("ID", 3) )
.CreateCriteria("Customer")
.Add(Restrictions.Eq("ID", 1) )
.List<User>();
Any other way (that works!) of doing it?
You can use alias
var userList = _session
.CreateCriteria(typeof(User), "u")
.CreateAlias("u.Role", "r")
.Add(Restrictions.Eq("r.ID", 3) )
.CreateAlias("u.Customer", "c")
.Add(Restrictions.Eq("c.ID", 1) )
.List<User>();
Hope it helps