I have a LINQ query that makes string search within a few tables. The query however is painfully slow on big tables. At my first attempt, I was getting a timeout. I was able to improve the performance a little. This is the first version of the code:
public ListResponse<UserDTO> GetUsers(FilterParameters filter)
{
var query = from user in _dbContext.Users
.Include(w => w.UserRoles).ThenInclude(u => u.Role)
join accountHolder in _dbContext.AccountHolders
.Include(c => c.OperationCountry)
.Include(x => x.Accounts)
.ThenInclude(x => x.Currency)
on user.Id equals accountHolder.ObjectId into aHolder
from a in aHolder.DefaultIfEmpty()
select new UserDTO
{
Id = user.Id,
FirstName = user.FirstName,
LastName = user.LastName,
Username = user.UserName,
Email = user.Email,
Roles = Mapper.Map<IList<RoleDTO>>(user.UserRoles.Select(i => i.Role)),
LastActivity = user.LastActivity,
CreatedAt = user.CreatedAt,
EmailConfirmed = user.EmailConfirmed,
AccountBalance = a.Accounts.Where(p => p.CurrencyId == a.OperationCountry.LocalCurrencyId).Single().Balance,
AccountReference = a.Accounts.Where(p => p.CurrencyId == a.OperationCountry.LocalCurrencyId).Single().AccountRef
};
// Apply search term
if (!IsNullOrEmpty(filter.SearchTerm))
query = query.Where(w =>
w.FirstName.Contains(filter.SearchTerm)
w.LastName.Contains(filter.SearchTerm) ||
w.Email.Contains(filter.SearchTerm) ||
w.AccountReference.Contains(filter.SearchTerm));
if (filter.ColumnFilters != null)
{
if (filter.ColumnFilters.ContainsKey("EmailConfirmed"))
{
var valueStr = filter.ColumnFilters["EmailConfirmed"];
if (bool.TryParse(valueStr, out var value))
query = query.Where(x => x.EmailConfirmed == value);
}
}
// Get total item count before pagination
var totalItemCount = query.Count();
// Apply pagination
query = query.ApplySortAndPagination(filter);
var userDtoList = query.ToList();
return new ListResponse<UserDTO>()
{
List = userDtoList,
TotalCount = totalItemCount
};
}
I suspected non-database code in the query (such as Single, and Mapping) was causing a slow query so I made an effort to get rid of them. I am still curious how to get a single Account without calling Single() inside the query. Here's the modified version.
public ListResponse<UserDTO> GetUsers(FilterParameters filter)
{
var query = from user in _dbContext.Users
.Include(w => w.UserRoles)
.ThenInclude(u => u.Role)
.Include(w => w.AccountHolder)
.ThenInclude(c => c.OperationCountry)
.Include(w => w.AccountHolder)
.ThenInclude(c => c.Accounts)
.ThenInclude(x => x.Currency)
select user;
if (!IsNullOrEmpty(filter.SearchTerm))
{
query = query.Where(w =>
w.FirstName.StartsWith(filter.SearchTerm) ||
w.LastName.StartsWith(filter.SearchTerm) ||
w.UserName.StartsWith(filter.SearchTerm) ||
w.AccountHolder.Accounts.Any(x => x.AccountRef.StartsWith(filter.SearchTerm)));
}
// total before pagination
var totalItemCount = query.Count();
// Nothing fancy, just OrderBy(filter.OrderBy).Skip(filter.Page).Take(filter.Length)
query = query.ApplySortAndPagination(filter);
userList = query.ToList() //To deal with "Single" calls below, this returns at most filter.Length records
var userDtoResult = (from user in query
select new UserDTO
{
Id = user.Id,
FirstName = user.FirstName,
LastName = user.LastName,
Username = user.UserName,
Email = user.Email,
Roles = Mapper.Map<IList<RoleDTO>>(user.UserRoles.Select(i => i.Role)),
LastActivity = user.LastActivity,
CreatedAt = user.CreatedAt,
EmailConfirmed = user.EmailConfirmed,
AccountBalance = user.AccountHolder.Accounts.Single(p => p.CurrencyId == user.AccountHolder.OperationCountry.LocalCurrencyId).Balance
AccountReference = user.AccountHolder.Accounts.Single(p => p.CurrencyId == user.AccountHolder.OperationCountry.LocalCurrencyId).AccountRef
}).ToList();
return new ListResponse<UserDTO>()
{
List = userDtoResult,
TotalCount = totalItemCount
};
}
The SQL query generated by this query runs slow too, whereas if I write a join query in SQL, it completes in a few hundred milliseconds. I am suspecting I am suffering from N+1 Query problem, but not sure since EF seems to generate a single query when I trace in the SQL Server Profiler.
This is the query generated by the Entity framework and runs in about 8 seconds when I run on the SSMS:
exec sp_executesql N'SELECT TOP(#__p_4) [w].[Id], [w].[AccessFailedCount], [w].[ConcurrencyStamp], [w].[CreatedAt], [w].[CreatedBy], [w].[DeletedAt], [w].[DeletedBy], [w].[DetailId], [w].[Email], [w].[EmailConfirmed], [w].[EmailConfirmedAt], [w].[FacebookId], [w].[FirstName], [w].[GoogleId], [w].[IsActive], [w].[IsDeleted], [w].[LastActivity], [w].[LastName], [w].[LockoutEnabled], [w].[LockoutEnd], [w].[NormalizedEmail], [w].[NormalizedUserName], [w].[Password], [w].[PasswordHash], [w].[PhoneNumber], [w].[PhoneNumberConfirmed], [w].[RoleId], [w].[SecurityStamp], [w].[TwoFactorEnabled], [w].[UpdatedAt], [w].[UpdatedBy], [w].[UserName], [w].[WorkflowId], [t].[Id], [t].[AccountHolderLevel], [t].[AccountHolderType], [t].[CreatedAt], [t].[CreatedBy], [t].[DeletedAt], [t].[DeletedBy], [t].[IsDeleted], [t].[ObjectId], [t].[OperationCountryId], [t].[UpdatedAt], [t].[UpdatedBy], [t0].[Id], [t0].[ContinentId], [t0].[CountryCode], [t0].[CreatedAt], [t0].[CreatedBy], [t0].[DeletedAt], [t0].[DeletedBy], [t0].[ISOCode2], [t0].[IsActive], [t0].[IsDeleted], [t0].[IsOperational], [t0].[LocalCurrencyId], [t0].[Name], [t0].[PhoneCode], [t0].[PostCodeProvider], [t0].[Regex], [t0].[SmsProvider], [t0].[UpdatedAt], [t0].[UpdatedBy]
FROM [Users] AS [w]
LEFT JOIN (
SELECT [a].[Id], [a].[AccountHolderLevel], [a].[AccountHolderType], [a].[CreatedAt], [a].[CreatedBy], [a].[DeletedAt], [a].[DeletedBy], [a].[IsDeleted], [a].[ObjectId], [a].[OperationCountryId], [a].[UpdatedAt], [a].[UpdatedBy]
FROM [AccountHolders] AS [a]
WHERE [a].[IsDeleted] = 0
) AS [t] ON [w].[Id] = [t].[ObjectId]
LEFT JOIN (
SELECT [c].[Id], [c].[ContinentId], [c].[CountryCode], [c].[CreatedAt], [c].[CreatedBy], [c].[DeletedAt], [c].[DeletedBy], [c].[ISOCode2], [c].[IsActive], [c].[IsDeleted], [c].[IsOperational], [c].[LocalCurrencyId], [c].[Name], [c].[PhoneCode], [c].[PostCodeProvider], [c].[Regex], [c].[SmsProvider], [c].[UpdatedAt], [c].[UpdatedBy]
FROM [Countries] AS [c]
WHERE [c].[IsDeleted] = 0
) AS [t0] ON [t].[OperationCountryId] = [t0].[Id]
WHERE ([w].[IsDeleted] = 0) AND ((((([w].[FirstName] LIKE #__filter_SearchTerm_0 + N''%'' AND (LEFT([w].[FirstName], LEN(#__filter_SearchTerm_0)) = #__filter_SearchTerm_0)) OR (#__filter_SearchTerm_0 = N'''')) OR (([w].[LastName] LIKE #__filter_SearchTerm_1 + N''%'' AND (LEFT([w].[LastName], LEN(#__filter_SearchTerm_1)) = #__filter_SearchTerm_1)) OR (#__filter_SearchTerm_1 = N''''))) OR (([w].[UserName] LIKE #__filter_SearchTerm_2 + N''%'' AND (LEFT([w].[UserName], LEN(#__filter_SearchTerm_2)) = #__filter_SearchTerm_2)) OR (#__filter_SearchTerm_2 = N''''))) OR EXISTS (
SELECT 1
FROM [Accounts] AS [x]
WHERE (([x].[IsDeleted] = 0) AND (([x].[AccountRef] LIKE #__filter_SearchTerm_3 + N''%'' AND (LEFT([x].[AccountRef], LEN(#__filter_SearchTerm_3)) = #__filter_SearchTerm_3)) OR (#__filter_SearchTerm_3 = N''''))) AND ([t].[Id] = [x].[AccountHolderId])))
ORDER BY [w].[LastActivity] DESC, [w].[Id], [t].[Id]',N'#__p_4 int,#__filter_SearchTerm_0 nvarchar(100),#__filter_SearchTerm_1 nvarchar(100),#__filter_SearchTerm_2 nvarchar(256),#__filter_SearchTerm_3 nvarchar(450)',#__p_4=10,#__filter_SearchTerm_0=N'james',#__filter_SearchTerm_1=N'james',#__filter_SearchTerm_2=N'james',#__filter_SearchTerm_3=N'james'
Finally this is my SQL query that returns whatever is necessary in less than 100 ms:
declare #searchTerm varchar(100) = '%james%'
select top 10
u.Id,
u.UserName,
u.FirstName,
u.LastName,
u.LastActivity,
u.CreatedAt,
a.Balance,
a.AccountRef,
ah.AccountHolderLevel,
u.Email,
r.Name
from Users u
join AccountHolders ah on ah.ObjectId = u.Id
join Accounts a on ah.Id = a.AccountHolderId
join UserRoles ur on ur.UserId = u.Id
join Roles r on r.Id = ur.RoleId
where FirstName like #searchTerm or LastName like #searchTerm or u.UserName like #searchTerm or FirstName + ' ' + LastName like #searchTerm or a.AccountRef like #searchTerm
and a.CurrencyId = ah.OperationCountryId
The columns I am searching are all indexed by the way, so that's not a problem. I know that the new EF-Core has many performance improvements. Unfortunately, I cannot update due to sheer number of breaking changes.
I am not sure splitting query into 2 (one for users and one for account) would work well, because there will be joins all over again. If I cannot find a solution using I plan converting my query to a view, but I want to do it as a last resort, since our convention is to use EF as much as possible. And I refuse to believe that EF does not have a solution. This is not actually a complex query at all and I am sure a fairly common use case.
So, what is the best way to optimize this query using EF-Core?
So, what is the best way to optimize this query using EF-Core?
Many things have changed in EF Core query pipeline since 2.1 (3.0, 3.1, 5.0 and now working on 6.0), but some general rules can be used, with the goal of getting rid of the client side query evaluation (which starting with 3.0 is not supported at all, so it's good to start preparing for the switch - support for 2.1 ends August this year).
The first would be to remove all these Include / ThenInclude. If the query is projecting the result in DTO without involving entity instances, then all these are redundant/not needed and removing them will ensure the query gets fully translated to SQL.
var query = _dbContext.Users.AsQueryable();
// Apply filters...
The next is the Roles collection. You must remove Mapper.Map call, otherwise it can't be translated. In general either use AutoMapper mappings and ProjectTo to fully handle the projection, or not use it at all (never put Map method calls inside query expression tree). According to your SQL, it should be something like this
Roles = user.UserRoles.Select(ur => ur.Role)
.Select(r => new RoleDTO { Name = r.Name })
.ToList(),
Actually EF Core will execute this as separate query (a behavior broken by "single query mode" in 3.x, and brought back optionally with 6.0 "split query mode"), so it is is important to have ToList() call at the end, otherwise you'll get N + 1 queries rather than 2.
Finally, the Single() call. It can be avoided by flattening the sub collection using correlated SelectMany, or its query syntax equivalent
from user in query
let ah = user.AccountHolder
from a in ah.Accounts
where a.CurrencyId == ah.OperationCountryId
The let statement is not mandatory, I've added it just for readability. Now you can use the range variables user, ah and a in the final select similar to table aliases in SQL.
Also since your SQL query doesn't really enforce single account match, there is no such enforcement in the LINQ query as well. If it was needed, then the equivalent of the Single can be achieved with SelectMany + Where + `Take(1), e.g.
from user in query
let ah = user.AccountHolder
from a in ah.Accounts
.Where(a => a.CurrencyId == ah.OperationCountryId)
.Take(1)
(a mixture of query and method syntax, but LINQ allows that)
So the final query would be something like this
from user in query
let ah = user.AccountHolder
from a in ah.Accounts
where a.CurrencyId == ah.OperationCountryId
select new //UserDTO
{
Id = user.Id,
FirstName = user.FirstName,
LastName = user.LastName,
Username = user.UserName,
Email = user.Email,
Roles = user.UserRoles.Select(ur => ur.Role)
.Select(r => new RoleDTO { Name = r.Name })
.ToList(),
LastActivity = user.LastActivity,
CreatedAt = user.CreatedAt,
EmailConfirmed = user.EmailConfirmed,
AccountBalance = a.Balance,
AccountReference = a.AccountRef
}
and should translate to very similar to the handcrafted SQL. And hopefully execute faster similar to it.
I am working on a project that uses TypeORM and PostgreSQL, I am trying to use the query builder to join on multiple conditions. Is there an easier/ more programmatic way to do this than having all the conditions within a string? For example, I want to build the following query to get friends for a user. In SQL this query looks like this (Note: inRel is short for incoming relationship and outRel is short for outgoing relationship)
-- SELECT FRIENDS FOR USER
select outRel."relatingToUserId"
from relationships outRel
inner join relationships inRel
on inRel."userId" = outRel."relatingToUserId"
and inRel."relatingToUserId" = outRel."userId"
and inRel."type" = 'FRIEND'
and outRel."type" = 'FRIEND'
where outRel."userId" = 'some_uuid_for_the_user';
In TypeORM I can accomplish the same result doing
const relationships = await this.createQueryBuilder()
.select('outRel.relatingToUserId')
.from(RelationshipEntity, 'outRel')
.innerJoin(
RelationshipEntity,
'inRel',
`
inRel.userId = outRel.relatingToUserId
AND inRel.relatingToUserId = outRel.userId
AND inRel.type = 'FRIEND'
AND inRel.type = outRel.type
`,
)
.where('outRel.userId = :userId', { userId })
.getMany();
However, I would expect that I should be able to do something more like
const relationships = await this.createQueryBuilder()
.select('outRel.relatingToUserId')
.from(RelationshipEntity, 'outRel')
.innerJoin(RelationshipEntity, 'inRel', 'inRel.userId = outRel.relatingToUserId')
.andWhere('inRel.relatingToUserId = outRel.userId')
.andWhere("inRel.type = 'FRIEND'")
.andWhere('inRel.type = outRel.type')
.where('outRel.userId = :userId', { userId })
.getMany();
But this does not return the same result. Is there a way to build this query more programmatically or am I stuck with a query string?
andWhere are used after .where. Try this:
const relationships = await this.createQueryBuilder()
.select('outRel.relatingToUserId')
.from(RelationshipEntity, 'outRel')
.innerJoin(RelationshipEntity, 'inRel', 'inRel.userId = outRel.relatingToUserId and inRel.relatingToUserId = outRel.userId and inRel.type = outRel.type')
.where('outRel.userId = :userId', { userId })
.andWhere('inRel.type = 'FRIEND'')
.getMany();
How can i dynamically change the selected columns in the generated sql query when using a linq expression?
Its a new session for each time the query is executed.
Even when I set the MapExp as null after first creation an then changing the bool value to false, it still generates the column in the sql query.
The code runs in a wpf application.
System.Linq.Expressions.Expression<Func<Entity, Model>> MapExp = x => new Model
{
Id=xId,
Count= LoadFormulaField ? x.Count: null,
...
};
var result = session.Query<Entity>().Select(MapExp))
Your problem seems to be the ternary-conditional as part of the expression which is causing the "Count" column to always be queried.
One option to avoid this could be:
var query = session.Query<Entity>();
IQueryable<Model> result = null;
if (LoadFormulaField)
{
result = query.Select(x => new Model
{
Id = x.Id,
Count = x.Count,
});
}
else
{
result = query.Select(x => new Model
{
Id = x.Id,
});
}
Which would get a little less ugly if you separate in a couple of methods I think.
How can I group the results of this query by wl.WishlistID?
var projected = (from wl in context.Wishlists.Where(x => x.UserId == 6013)
from wli in wl.WishlistItems
select new wishListProjection
{
wlId = wl.WishlistID,
wlUserId = (int)wl.UserId,
wlDesc = wl.description,
wlTitle = wl.Title,
wliWishlistItemID = wli.WishlistItemID,
wliQtyWanted = (int)wli.QtyWanted,
wliPriceWhenAdded = wli.PriceWhenAdded,
wliDateAdded = (DateTime)wli.DateAdded,
wliQtyBought = (int)wli.QtyBought,
}).ToList();
This returns the results I want, but I want to be able to iterate over them in the view without repeating the parent-level Wishlist object.
I've tried adding the line:
group wl by wl.WishlistID into g
But I don't seem to be able to access any properties by using g.[propertyname]
This grouping achieves more or less what I want, but I want to convert the results into a new or anonymous type, rather than return the entire object.
var results = context.WishlistItems.GroupBy(x => x.Wishlist).
Select(group => new { wl = group.Key, items = group.ToList() }).ToList();
You cannot access the properties of g because when you do the grouping:
group wl by wl.WishlistID into g
g is of type IGrouping<typeof(wl.WishlistID),typeof(wl)>, which is essentially a collection of all wl's with the same key wl.WishlistID. In other words, you cannot access the properties of g because g is not a single entity, it is a collection of those entities.
For your second grouping you said you would like to create an anonymous type instead of the entire object. You can do this by doing the selection first and then grouping:
var results = context.WishlistItems
.Select(x => new { })
.GroupBy(x => x.PropertyOfProjection)
.Select(group => new { wl = group.Key, items = group.ToList() }).ToList();
Or, using a nested sub query in your first example:
var projected = (from x in
(from wl in context.Wishlists.Where(x => x.UserId == 6013)
from wli in wl.WishlistItems
select new wishListProjection
{
wlId = wl.WishlistID,
wlUserId = (int)wl.UserId,
wlDesc = wl.description,
wlTitle = wl.Title,
wliWishlistItemID = wli.WishlistItemID,
wliQtyWanted = (int)wli.QtyWanted,
wliPriceWhenAdded = wli.PriceWhenAdded,
wliDateAdded = (DateTime)wli.DateAdded,
wliQtyBought = (int)wli.QtyBought,
})
group x by w.wlId into g).ToList();
I'm not sure what you mean by iterate without repeating the parent-level Wishlist object because whenever you create a grouping in Linq you will still have to have a nested foreach like:
foreach (var x in grouping)
{
x.Key;
foreach (var y in x)
{
y.Property;
}
}
How to convert the following sql query to lambda expression?
select cg.Code, ci.ChangeType, SUM(oc.Value) from OtherCharges oc
left join changeitems ci on oc.ChangeItemKey = ci.ChangeItemKey
left join ChangeGroups cg on ci.ChangeGroupKey = cg.ChangeGroupKey
where OtherKey = 'AB235A00-FEB2-4C4F-B0F9-3239FD127A8F'
group by cg.Code, ci.ChangeType
order by cg.Code, ci.ChangeType
Assuming you already have .NET domain types for your tables:
IQueryable<OtherCharges> otherCharges = ...
Guid otherKey = ...
var query = otherCharges.Where(oc => oc.OtherKey == otherKey)
.Select(oc => new { oc.ChangeItem, oc.Value })
.GroupBy(t => new { t.ChangeItem.ChangeGroup.Code, t.ChangeItem.ChangeType })
.OrderBy(g => g.Key.Code)
.ThenBy(g => g.Key.ChangeType)
// note that if Code is a non-nullable type you'll want to cast it to int? at some
// point so that when pulled into memory EF won't complain that you can't cast
// null to a non-nullable type. I expect that Code could sometimes be null here
// do to your use of LEFT OUTER JOIN in the T-SQL
.Select(g => new { g.Key.Code, g.Key.ChangeType, Sum = g.Sum(t => t.Value) });
var inMemoryResult = query.ToList();
Note that I'm using OtherCharge.ChangeItem and ChangeItem.ChangeGroup here. These are association properties and need to be set up as part of your model (e. g. using fluent configuration for EF code first).