I have a LINQ query that makes string search within a few tables. The query however is painfully slow on big tables. At my first attempt, I was getting a timeout. I was able to improve the performance a little. This is the first version of the code:
public ListResponse<UserDTO> GetUsers(FilterParameters filter)
{
var query = from user in _dbContext.Users
.Include(w => w.UserRoles).ThenInclude(u => u.Role)
join accountHolder in _dbContext.AccountHolders
.Include(c => c.OperationCountry)
.Include(x => x.Accounts)
.ThenInclude(x => x.Currency)
on user.Id equals accountHolder.ObjectId into aHolder
from a in aHolder.DefaultIfEmpty()
select new UserDTO
{
Id = user.Id,
FirstName = user.FirstName,
LastName = user.LastName,
Username = user.UserName,
Email = user.Email,
Roles = Mapper.Map<IList<RoleDTO>>(user.UserRoles.Select(i => i.Role)),
LastActivity = user.LastActivity,
CreatedAt = user.CreatedAt,
EmailConfirmed = user.EmailConfirmed,
AccountBalance = a.Accounts.Where(p => p.CurrencyId == a.OperationCountry.LocalCurrencyId).Single().Balance,
AccountReference = a.Accounts.Where(p => p.CurrencyId == a.OperationCountry.LocalCurrencyId).Single().AccountRef
};
// Apply search term
if (!IsNullOrEmpty(filter.SearchTerm))
query = query.Where(w =>
w.FirstName.Contains(filter.SearchTerm)
w.LastName.Contains(filter.SearchTerm) ||
w.Email.Contains(filter.SearchTerm) ||
w.AccountReference.Contains(filter.SearchTerm));
if (filter.ColumnFilters != null)
{
if (filter.ColumnFilters.ContainsKey("EmailConfirmed"))
{
var valueStr = filter.ColumnFilters["EmailConfirmed"];
if (bool.TryParse(valueStr, out var value))
query = query.Where(x => x.EmailConfirmed == value);
}
}
// Get total item count before pagination
var totalItemCount = query.Count();
// Apply pagination
query = query.ApplySortAndPagination(filter);
var userDtoList = query.ToList();
return new ListResponse<UserDTO>()
{
List = userDtoList,
TotalCount = totalItemCount
};
}
I suspected non-database code in the query (such as Single, and Mapping) was causing a slow query so I made an effort to get rid of them. I am still curious how to get a single Account without calling Single() inside the query. Here's the modified version.
public ListResponse<UserDTO> GetUsers(FilterParameters filter)
{
var query = from user in _dbContext.Users
.Include(w => w.UserRoles)
.ThenInclude(u => u.Role)
.Include(w => w.AccountHolder)
.ThenInclude(c => c.OperationCountry)
.Include(w => w.AccountHolder)
.ThenInclude(c => c.Accounts)
.ThenInclude(x => x.Currency)
select user;
if (!IsNullOrEmpty(filter.SearchTerm))
{
query = query.Where(w =>
w.FirstName.StartsWith(filter.SearchTerm) ||
w.LastName.StartsWith(filter.SearchTerm) ||
w.UserName.StartsWith(filter.SearchTerm) ||
w.AccountHolder.Accounts.Any(x => x.AccountRef.StartsWith(filter.SearchTerm)));
}
// total before pagination
var totalItemCount = query.Count();
// Nothing fancy, just OrderBy(filter.OrderBy).Skip(filter.Page).Take(filter.Length)
query = query.ApplySortAndPagination(filter);
userList = query.ToList() //To deal with "Single" calls below, this returns at most filter.Length records
var userDtoResult = (from user in query
select new UserDTO
{
Id = user.Id,
FirstName = user.FirstName,
LastName = user.LastName,
Username = user.UserName,
Email = user.Email,
Roles = Mapper.Map<IList<RoleDTO>>(user.UserRoles.Select(i => i.Role)),
LastActivity = user.LastActivity,
CreatedAt = user.CreatedAt,
EmailConfirmed = user.EmailConfirmed,
AccountBalance = user.AccountHolder.Accounts.Single(p => p.CurrencyId == user.AccountHolder.OperationCountry.LocalCurrencyId).Balance
AccountReference = user.AccountHolder.Accounts.Single(p => p.CurrencyId == user.AccountHolder.OperationCountry.LocalCurrencyId).AccountRef
}).ToList();
return new ListResponse<UserDTO>()
{
List = userDtoResult,
TotalCount = totalItemCount
};
}
The SQL query generated by this query runs slow too, whereas if I write a join query in SQL, it completes in a few hundred milliseconds. I am suspecting I am suffering from N+1 Query problem, but not sure since EF seems to generate a single query when I trace in the SQL Server Profiler.
This is the query generated by the Entity framework and runs in about 8 seconds when I run on the SSMS:
exec sp_executesql N'SELECT TOP(#__p_4) [w].[Id], [w].[AccessFailedCount], [w].[ConcurrencyStamp], [w].[CreatedAt], [w].[CreatedBy], [w].[DeletedAt], [w].[DeletedBy], [w].[DetailId], [w].[Email], [w].[EmailConfirmed], [w].[EmailConfirmedAt], [w].[FacebookId], [w].[FirstName], [w].[GoogleId], [w].[IsActive], [w].[IsDeleted], [w].[LastActivity], [w].[LastName], [w].[LockoutEnabled], [w].[LockoutEnd], [w].[NormalizedEmail], [w].[NormalizedUserName], [w].[Password], [w].[PasswordHash], [w].[PhoneNumber], [w].[PhoneNumberConfirmed], [w].[RoleId], [w].[SecurityStamp], [w].[TwoFactorEnabled], [w].[UpdatedAt], [w].[UpdatedBy], [w].[UserName], [w].[WorkflowId], [t].[Id], [t].[AccountHolderLevel], [t].[AccountHolderType], [t].[CreatedAt], [t].[CreatedBy], [t].[DeletedAt], [t].[DeletedBy], [t].[IsDeleted], [t].[ObjectId], [t].[OperationCountryId], [t].[UpdatedAt], [t].[UpdatedBy], [t0].[Id], [t0].[ContinentId], [t0].[CountryCode], [t0].[CreatedAt], [t0].[CreatedBy], [t0].[DeletedAt], [t0].[DeletedBy], [t0].[ISOCode2], [t0].[IsActive], [t0].[IsDeleted], [t0].[IsOperational], [t0].[LocalCurrencyId], [t0].[Name], [t0].[PhoneCode], [t0].[PostCodeProvider], [t0].[Regex], [t0].[SmsProvider], [t0].[UpdatedAt], [t0].[UpdatedBy]
FROM [Users] AS [w]
LEFT JOIN (
SELECT [a].[Id], [a].[AccountHolderLevel], [a].[AccountHolderType], [a].[CreatedAt], [a].[CreatedBy], [a].[DeletedAt], [a].[DeletedBy], [a].[IsDeleted], [a].[ObjectId], [a].[OperationCountryId], [a].[UpdatedAt], [a].[UpdatedBy]
FROM [AccountHolders] AS [a]
WHERE [a].[IsDeleted] = 0
) AS [t] ON [w].[Id] = [t].[ObjectId]
LEFT JOIN (
SELECT [c].[Id], [c].[ContinentId], [c].[CountryCode], [c].[CreatedAt], [c].[CreatedBy], [c].[DeletedAt], [c].[DeletedBy], [c].[ISOCode2], [c].[IsActive], [c].[IsDeleted], [c].[IsOperational], [c].[LocalCurrencyId], [c].[Name], [c].[PhoneCode], [c].[PostCodeProvider], [c].[Regex], [c].[SmsProvider], [c].[UpdatedAt], [c].[UpdatedBy]
FROM [Countries] AS [c]
WHERE [c].[IsDeleted] = 0
) AS [t0] ON [t].[OperationCountryId] = [t0].[Id]
WHERE ([w].[IsDeleted] = 0) AND ((((([w].[FirstName] LIKE #__filter_SearchTerm_0 + N''%'' AND (LEFT([w].[FirstName], LEN(#__filter_SearchTerm_0)) = #__filter_SearchTerm_0)) OR (#__filter_SearchTerm_0 = N'''')) OR (([w].[LastName] LIKE #__filter_SearchTerm_1 + N''%'' AND (LEFT([w].[LastName], LEN(#__filter_SearchTerm_1)) = #__filter_SearchTerm_1)) OR (#__filter_SearchTerm_1 = N''''))) OR (([w].[UserName] LIKE #__filter_SearchTerm_2 + N''%'' AND (LEFT([w].[UserName], LEN(#__filter_SearchTerm_2)) = #__filter_SearchTerm_2)) OR (#__filter_SearchTerm_2 = N''''))) OR EXISTS (
SELECT 1
FROM [Accounts] AS [x]
WHERE (([x].[IsDeleted] = 0) AND (([x].[AccountRef] LIKE #__filter_SearchTerm_3 + N''%'' AND (LEFT([x].[AccountRef], LEN(#__filter_SearchTerm_3)) = #__filter_SearchTerm_3)) OR (#__filter_SearchTerm_3 = N''''))) AND ([t].[Id] = [x].[AccountHolderId])))
ORDER BY [w].[LastActivity] DESC, [w].[Id], [t].[Id]',N'#__p_4 int,#__filter_SearchTerm_0 nvarchar(100),#__filter_SearchTerm_1 nvarchar(100),#__filter_SearchTerm_2 nvarchar(256),#__filter_SearchTerm_3 nvarchar(450)',#__p_4=10,#__filter_SearchTerm_0=N'james',#__filter_SearchTerm_1=N'james',#__filter_SearchTerm_2=N'james',#__filter_SearchTerm_3=N'james'
Finally this is my SQL query that returns whatever is necessary in less than 100 ms:
declare #searchTerm varchar(100) = '%james%'
select top 10
u.Id,
u.UserName,
u.FirstName,
u.LastName,
u.LastActivity,
u.CreatedAt,
a.Balance,
a.AccountRef,
ah.AccountHolderLevel,
u.Email,
r.Name
from Users u
join AccountHolders ah on ah.ObjectId = u.Id
join Accounts a on ah.Id = a.AccountHolderId
join UserRoles ur on ur.UserId = u.Id
join Roles r on r.Id = ur.RoleId
where FirstName like #searchTerm or LastName like #searchTerm or u.UserName like #searchTerm or FirstName + ' ' + LastName like #searchTerm or a.AccountRef like #searchTerm
and a.CurrencyId = ah.OperationCountryId
The columns I am searching are all indexed by the way, so that's not a problem. I know that the new EF-Core has many performance improvements. Unfortunately, I cannot update due to sheer number of breaking changes.
I am not sure splitting query into 2 (one for users and one for account) would work well, because there will be joins all over again. If I cannot find a solution using I plan converting my query to a view, but I want to do it as a last resort, since our convention is to use EF as much as possible. And I refuse to believe that EF does not have a solution. This is not actually a complex query at all and I am sure a fairly common use case.
So, what is the best way to optimize this query using EF-Core?
So, what is the best way to optimize this query using EF-Core?
Many things have changed in EF Core query pipeline since 2.1 (3.0, 3.1, 5.0 and now working on 6.0), but some general rules can be used, with the goal of getting rid of the client side query evaluation (which starting with 3.0 is not supported at all, so it's good to start preparing for the switch - support for 2.1 ends August this year).
The first would be to remove all these Include / ThenInclude. If the query is projecting the result in DTO without involving entity instances, then all these are redundant/not needed and removing them will ensure the query gets fully translated to SQL.
var query = _dbContext.Users.AsQueryable();
// Apply filters...
The next is the Roles collection. You must remove Mapper.Map call, otherwise it can't be translated. In general either use AutoMapper mappings and ProjectTo to fully handle the projection, or not use it at all (never put Map method calls inside query expression tree). According to your SQL, it should be something like this
Roles = user.UserRoles.Select(ur => ur.Role)
.Select(r => new RoleDTO { Name = r.Name })
.ToList(),
Actually EF Core will execute this as separate query (a behavior broken by "single query mode" in 3.x, and brought back optionally with 6.0 "split query mode"), so it is is important to have ToList() call at the end, otherwise you'll get N + 1 queries rather than 2.
Finally, the Single() call. It can be avoided by flattening the sub collection using correlated SelectMany, or its query syntax equivalent
from user in query
let ah = user.AccountHolder
from a in ah.Accounts
where a.CurrencyId == ah.OperationCountryId
The let statement is not mandatory, I've added it just for readability. Now you can use the range variables user, ah and a in the final select similar to table aliases in SQL.
Also since your SQL query doesn't really enforce single account match, there is no such enforcement in the LINQ query as well. If it was needed, then the equivalent of the Single can be achieved with SelectMany + Where + `Take(1), e.g.
from user in query
let ah = user.AccountHolder
from a in ah.Accounts
.Where(a => a.CurrencyId == ah.OperationCountryId)
.Take(1)
(a mixture of query and method syntax, but LINQ allows that)
So the final query would be something like this
from user in query
let ah = user.AccountHolder
from a in ah.Accounts
where a.CurrencyId == ah.OperationCountryId
select new //UserDTO
{
Id = user.Id,
FirstName = user.FirstName,
LastName = user.LastName,
Username = user.UserName,
Email = user.Email,
Roles = user.UserRoles.Select(ur => ur.Role)
.Select(r => new RoleDTO { Name = r.Name })
.ToList(),
LastActivity = user.LastActivity,
CreatedAt = user.CreatedAt,
EmailConfirmed = user.EmailConfirmed,
AccountBalance = a.Balance,
AccountReference = a.AccountRef
}
and should translate to very similar to the handcrafted SQL. And hopefully execute faster similar to it.
I have a situation where two tables should be joined with multiple columns with or condition. Here, I have a sample of sql query but i was not able to convert it into linq query.
select cm.* from Customer cm
inner join #temp tmp
on cm.CustomerCode = tmp.NewNLKNo or cm.OldAcNo = tmp.OldNLKNo
This is how i have write linq query
await (from cm in Context.CustomerMaster
join li in list.PortalCustomerDetailViewModel
on new { OldNLKNo = cm.OldAcNo, NewNLKNo = cm.CustomerCode } equals new { OldNLKNo = li.OldNLKNo, NewNLKNo = li.NewNLKNo }
select new CustomerInfoViewModel
{
CustomerId = cm.Id,
CustomerCode = cm.CustomerCode,
CustomerFullName = cm.CustomerFullName,
OldCustomerCode = cm.OldCustomerCode,
IsCorporateCustomer = cm.IsCorporateCustomer
}).ToListAsync();
But this query doesn't returns as expected. How do I convert this sql query into linq.
Thank you
You didn't tell if list.PortalCustomerDetailViewModel is some information in the database, or in your local process. It seems that this is in your local process, your query will have to transfer it to the database (maybe that is why it is Tmp in your SQL?)
Requirement: give me all properties of a CustomerMaster for all CustomerMasters where exists at least one PortalCustomerDetailViewModel where
customerMaster.CustomerCode == portalCustomerDetailViewModel.NewNLKNo
|| customerMaster.OldAcNo == portalCustomerDetailViewModel.OldNLKNo
You can't use a normal Join, because a Join works with an AND, you want to work with OR
What you could do, is Select all CustomerMasters where there is any PortalCustomerDetailViewModel that fulfills the provided OR:
I only transfer those properties of list.PortalCustomerDetailViewModel to the database that I need to use in the OR expression:
var checkProperties = list.PortalCustomerDetailViewModel
.Select(portalCustomerDetail => new
{
NewNlkNo = portalCustomerDetail.NewNlkNo,
OldNLKNo = portalCustomerDetail.OldNLKNo,
});
var result = dbContext.CustomerMasters.Where(customerMaster =>
checkProperties.Where(checkProperty =>
customerMaster.CustomerCode == checkProperty.NewNLKNo
|| customerMaster.OldAcNo == checkProperty.OldNLKNo)).Any()))
.Select(customerMaster => new CustomerInfoViewModel
{
Id = customerMaster.Id,
Name = customerMaster.Name,
...
});
In words: from each portalCustomerDetail in list.PortalCustomerDetailViewModel, extract the properties NewNKLNo and OldNLKNo.
Then from the table of CustomerMasters, keep only those customerMasters that have at least one portalCustomerDetail with the properties as described in the OR statement.
From every remaining CustomerMasters, create one new CustomerInfoViewModel containing properties ...
select cm.* from Customer cm
inner join #temp tmp
on cm.CustomerCode = tmp.NewNLKNo or cm.OldAcNo = tmp.OldNLKNo
You don't have to use the join syntax. Adding the predicates in a where clause could get the same result. Try to use the following code:
await (from cm in Context.CustomerMaster
from li in list.PortalCustomerDetailViewModel
where cm.CustomerCode == li.NewNLKNo || cm.OldAcNo = li.OldNLKNo
select new CustomerInfoViewModel
{
CustomerId = cm.Id,
CustomerCode = cm.CustomerCode,
CustomerFullName = cm.CustomerFullName,
OldCustomerCode = cm.OldCustomerCode,
IsCorporateCustomer = cm.IsCorporateCustomer
}).ToListAsync();
var result=_db.Customer
.groupjoin(_db.#temp ,jc=>jc.CustomerCode,c=> c.NewNLKNo,(jc,c)=>{jc,c=c.firstordefault()})
.groupjoin(_db.#temp ,jc2=>jc2.OldAcNo,c2=> c2.OldNLKNo,(jc2,c2)=>{jc2,c2=c2.firstordefault()})
.select(x=> new{
//as you want
}).distinct().tolist();
I am converting a SQL query to LINQ that creates a left join with 1-to-1 mapping, and it has to be in Method Syntax. I have been pulling off my hair trying to accomplish this to no veil. I can do it in Lambda Syntax. Below is the example query I am trying to run. They are not actual code. Would someone point out what I am doing wrong?
SQL:
SELECT item.*, item_status.*
FROM item
LEFT JOIN item_status
ON item.ID = item_status.itemID
AND item_status.FLAGGED = true
WHERE item.published_date > "2008-06-19"
LINQ:
var linq_query = (
from selected_item in item
join selected_item_status in item_status
on selected_item.ID equals item_status.itemID into joined
from item_status in joined.DefaultIfEmpty()
where item_status.FLAGGED = true
select new {selected_item, selected_item_status}).ToList();
The join ... into becomes a GroupJoin and the second from becomes a SelectMany:
var linq_query = Item
.GroupJoin(
item_status.Where(x => x.selected_item_status.FLAGGED), // EDIT: Where clause moved here.
selected_item => selected_item.ID,
selected_item_status => selected_item_status.itemID,
(selected_item, joined) => new
{
selected_item,
statuses = joined.DefaultWithEmpty(),
})
.SelectMany(x => x.statuses.Select(selected_item_status => new
{
x.selected_item,
selected_item_status,
}))
// EDIT: Removed where clause.
.ToList();
It looks like the Where makes the left outer join unnecessary, as null statuses will be filtered out anyway.
EDIT: No, upon reviewing the SQL it looks like your LINQ query is slightly incorrect. It should be:
var linq_query = (
from selected_item in item
join selected_item_status
in (
from status in item_status
where status.FLAGGED
select status)
on selected_item.ID equals item_status.itemID into joined
from item_status in joined.DefaultIfEmpty()
select new {selected_item, selected_item_status}).ToList();
Hi i'm try to debug some code but can't work out what it's doing.
// For reference, the following is roughly equivalent to:
// select p.*
// from CTBEquitiesFiles as o inner join CTBEquitiesDetailStaging as p
// on o.ID = p.CTBEquitiesFiles.ID
// where o.SEFileDate = fileDate and o.SEFileType = 'W'
mdovar depositRows = fileRepo.Query(o => o.SEFileDate == fileDate && o.SEFileType.Equals("D"))
.Join<SE.TradeDetailFile, SE.TradeDetailStaging, int, SE.TradeDetailStaging>(
detailRepo.Get(),
o => o.ID,
p => p.CTBEquitiesFiles.ID,
(o, p) => p);
I'm fairly sure it's not doing what I want. I want to get all the rows from CTBEquitiesFilesDetailsStaging where the SEFileID column is the same as the ID column in CTBEquitiesFiles.
Thank you
If you want to see what queries are being issued to the database, you can use the sql profiler, or set the DataContext.Log property.
My LINQ query contains the following Group By statement:
Group p By Key = New With { _
.Latitude = p.Address.GeoLocations.FirstOrDefault(Function(g) New String() {"ADDRESS", "POINT"}.Contains(g.Granularity)).Latitude, _
.Longitude = p.Address.GeoLocations.FirstOrDefault(Function(g) New String() {"ADDRESS", "POINT"}.Contains(g.Granularity)).Longitude}
The query works, but here is the SQL that the clause above produces
SELECT [t6].[Latitude]
FROM (
SELECT TOP (1) [t5].[Latitude]
FROM [dbo].[GeoLocations] AS [t5]
WHERE ([t5].[Granularity] IN (#p0, #p1)) AND ([t5].[AddressId] = [t2].[Addr_AddressId])
) AS [t6]
) AS [value], (
SELECT [t8].[Longitude]
FROM (
SELECT TOP (1) [t7].[Longitude]
FROM [dbo].[GeoLocations] AS [t7]
WHERE ([t7].[Granularity] IN (#p2, #p3)) AND ([t7].[AddressId] = [t2].[Addr_AddressId])
) AS [t8]
) AS [value2]
I am not a SQL expert, but it looks to me that this is rather suboptimal translation. This should really be one query that selects Latitide and Longitude from the first record. Perhaps SQL Server Optimizer will take care of this. But is there a way to nudge Linq to generate a leaner SQL statement to begin with?
I tried the following, too..
Group p By Key = p.Address.GeoLocations.Where(Function(g) New String() {"ADDRESS", "POINT"}.Contains(g.Granularity)). _
Select(Function(g) New With {.Latitude = g.Latitude, .Longitude = g.Longitude}).FirstOrDefault
but this produced an error: "A group by expression can only contain non-constant scalars that are comparable by the server."
Sorry to reply in c#...
Here's what you have, translated to c#:
List<string> params = new List<string>()
{ "Address", "Point" };
from p in people
group p by new {
Latitude = p.Address.GeoLocations
.FirstOrDefault(g => params.Contains(g.Granularity)).Latitude,
Longitude = p.Address.GeoLocations
.FirstOrDefault(g => params.Contains(g.Granularity)).Longitude
};
Here's a rewrite, using the let keyword.
from p in people
let loc = p.Address.GeoLocations
.FirstOrDefault(g => params.Contains(g.Granularity))
group p by new
{
Latitude = loc.Latitude,
Longitude = loc.Longitude
};