Entity Framework and dynamic order by statements - sql

I have been struggling to get this working. I wish to have an EF statement take in a column to order by. My original statement was this:
var Query = from P in DbContext.People
where P.BusinessUnits.Any(BU =>BU.BusinessUnitID == businessUnitId)
orderby P.LastName
select P;
And I changed this to the following:
var Query = from P in DbContext.People
where P.BusinessUnits.Any(BU =>BU.BusinessUnitID == businessUnitId)
orderby sortField
select P;
Where sortField is the column we wish to sort on, and is a string i.e. LastName. However, it does not appear to work, it does no sorting, and the outputted SQL string is completely wrong. Anyone got this working before?

you could try passing in an expression to your method with the following type:
Expression<Func<Person, object>> expr = p => p.LastName;
and then using linq extensions instead of linq expressions...
var Query =
DbContext.People
.Where(P => P.BusinessUnits.Any(BU =>BU.BusinessUnitID == businessUnitId))
.OrderBy(expr)
.ToList();

Your sort does not work because you are sorting on a string literal. It is not illegal, but it is not particularly useful either. You need to provide a sorting field through the API of IQueryable<T>, for example, like this:
var q = from P in DbContext.People
where P.BusinessUnits.Any(BU =>BU.BusinessUnitID == businessUnitId)
orderby P.LastName
select P;
if ("sortField".Equals("FirstName"))
q = q.OrderBy(p => p.FirstName);
else if ("sortField".Equals("LastName"))
q = q.OrderBy(p => p.LastName);
else if ("sortField".Equals("Dob"))
q = q.OrderBy(p => p.Dob);

Related

How can I optimize slow (not-so) complex queries in Entity Framework Core 2.1

I have a LINQ query that makes string search within a few tables. The query however is painfully slow on big tables. At my first attempt, I was getting a timeout. I was able to improve the performance a little. This is the first version of the code:
public ListResponse<UserDTO> GetUsers(FilterParameters filter)
{
var query = from user in _dbContext.Users
.Include(w => w.UserRoles).ThenInclude(u => u.Role)
join accountHolder in _dbContext.AccountHolders
.Include(c => c.OperationCountry)
.Include(x => x.Accounts)
.ThenInclude(x => x.Currency)
on user.Id equals accountHolder.ObjectId into aHolder
from a in aHolder.DefaultIfEmpty()
select new UserDTO
{
Id = user.Id,
FirstName = user.FirstName,
LastName = user.LastName,
Username = user.UserName,
Email = user.Email,
Roles = Mapper.Map<IList<RoleDTO>>(user.UserRoles.Select(i => i.Role)),
LastActivity = user.LastActivity,
CreatedAt = user.CreatedAt,
EmailConfirmed = user.EmailConfirmed,
AccountBalance = a.Accounts.Where(p => p.CurrencyId == a.OperationCountry.LocalCurrencyId).Single().Balance,
AccountReference = a.Accounts.Where(p => p.CurrencyId == a.OperationCountry.LocalCurrencyId).Single().AccountRef
};
// Apply search term
if (!IsNullOrEmpty(filter.SearchTerm))
query = query.Where(w =>
w.FirstName.Contains(filter.SearchTerm)
w.LastName.Contains(filter.SearchTerm) ||
w.Email.Contains(filter.SearchTerm) ||
w.AccountReference.Contains(filter.SearchTerm));
if (filter.ColumnFilters != null)
{
if (filter.ColumnFilters.ContainsKey("EmailConfirmed"))
{
var valueStr = filter.ColumnFilters["EmailConfirmed"];
if (bool.TryParse(valueStr, out var value))
query = query.Where(x => x.EmailConfirmed == value);
}
}
// Get total item count before pagination
var totalItemCount = query.Count();
// Apply pagination
query = query.ApplySortAndPagination(filter);
var userDtoList = query.ToList();
return new ListResponse<UserDTO>()
{
List = userDtoList,
TotalCount = totalItemCount
};
}
I suspected non-database code in the query (such as Single, and Mapping) was causing a slow query so I made an effort to get rid of them. I am still curious how to get a single Account without calling Single() inside the query. Here's the modified version.
public ListResponse<UserDTO> GetUsers(FilterParameters filter)
{
var query = from user in _dbContext.Users
.Include(w => w.UserRoles)
.ThenInclude(u => u.Role)
.Include(w => w.AccountHolder)
.ThenInclude(c => c.OperationCountry)
.Include(w => w.AccountHolder)
.ThenInclude(c => c.Accounts)
.ThenInclude(x => x.Currency)
select user;
if (!IsNullOrEmpty(filter.SearchTerm))
{
query = query.Where(w =>
w.FirstName.StartsWith(filter.SearchTerm) ||
w.LastName.StartsWith(filter.SearchTerm) ||
w.UserName.StartsWith(filter.SearchTerm) ||
w.AccountHolder.Accounts.Any(x => x.AccountRef.StartsWith(filter.SearchTerm)));
}
// total before pagination
var totalItemCount = query.Count();
// Nothing fancy, just OrderBy(filter.OrderBy).Skip(filter.Page).Take(filter.Length)
query = query.ApplySortAndPagination(filter);
userList = query.ToList() //To deal with "Single" calls below, this returns at most filter.Length records
var userDtoResult = (from user in query
select new UserDTO
{
Id = user.Id,
FirstName = user.FirstName,
LastName = user.LastName,
Username = user.UserName,
Email = user.Email,
Roles = Mapper.Map<IList<RoleDTO>>(user.UserRoles.Select(i => i.Role)),
LastActivity = user.LastActivity,
CreatedAt = user.CreatedAt,
EmailConfirmed = user.EmailConfirmed,
AccountBalance = user.AccountHolder.Accounts.Single(p => p.CurrencyId == user.AccountHolder.OperationCountry.LocalCurrencyId).Balance
AccountReference = user.AccountHolder.Accounts.Single(p => p.CurrencyId == user.AccountHolder.OperationCountry.LocalCurrencyId).AccountRef
}).ToList();
return new ListResponse<UserDTO>()
{
List = userDtoResult,
TotalCount = totalItemCount
};
}
The SQL query generated by this query runs slow too, whereas if I write a join query in SQL, it completes in a few hundred milliseconds. I am suspecting I am suffering from N+1 Query problem, but not sure since EF seems to generate a single query when I trace in the SQL Server Profiler.
This is the query generated by the Entity framework and runs in about 8 seconds when I run on the SSMS:
exec sp_executesql N'SELECT TOP(#__p_4) [w].[Id], [w].[AccessFailedCount], [w].[ConcurrencyStamp], [w].[CreatedAt], [w].[CreatedBy], [w].[DeletedAt], [w].[DeletedBy], [w].[DetailId], [w].[Email], [w].[EmailConfirmed], [w].[EmailConfirmedAt], [w].[FacebookId], [w].[FirstName], [w].[GoogleId], [w].[IsActive], [w].[IsDeleted], [w].[LastActivity], [w].[LastName], [w].[LockoutEnabled], [w].[LockoutEnd], [w].[NormalizedEmail], [w].[NormalizedUserName], [w].[Password], [w].[PasswordHash], [w].[PhoneNumber], [w].[PhoneNumberConfirmed], [w].[RoleId], [w].[SecurityStamp], [w].[TwoFactorEnabled], [w].[UpdatedAt], [w].[UpdatedBy], [w].[UserName], [w].[WorkflowId], [t].[Id], [t].[AccountHolderLevel], [t].[AccountHolderType], [t].[CreatedAt], [t].[CreatedBy], [t].[DeletedAt], [t].[DeletedBy], [t].[IsDeleted], [t].[ObjectId], [t].[OperationCountryId], [t].[UpdatedAt], [t].[UpdatedBy], [t0].[Id], [t0].[ContinentId], [t0].[CountryCode], [t0].[CreatedAt], [t0].[CreatedBy], [t0].[DeletedAt], [t0].[DeletedBy], [t0].[ISOCode2], [t0].[IsActive], [t0].[IsDeleted], [t0].[IsOperational], [t0].[LocalCurrencyId], [t0].[Name], [t0].[PhoneCode], [t0].[PostCodeProvider], [t0].[Regex], [t0].[SmsProvider], [t0].[UpdatedAt], [t0].[UpdatedBy]
FROM [Users] AS [w]
LEFT JOIN (
SELECT [a].[Id], [a].[AccountHolderLevel], [a].[AccountHolderType], [a].[CreatedAt], [a].[CreatedBy], [a].[DeletedAt], [a].[DeletedBy], [a].[IsDeleted], [a].[ObjectId], [a].[OperationCountryId], [a].[UpdatedAt], [a].[UpdatedBy]
FROM [AccountHolders] AS [a]
WHERE [a].[IsDeleted] = 0
) AS [t] ON [w].[Id] = [t].[ObjectId]
LEFT JOIN (
SELECT [c].[Id], [c].[ContinentId], [c].[CountryCode], [c].[CreatedAt], [c].[CreatedBy], [c].[DeletedAt], [c].[DeletedBy], [c].[ISOCode2], [c].[IsActive], [c].[IsDeleted], [c].[IsOperational], [c].[LocalCurrencyId], [c].[Name], [c].[PhoneCode], [c].[PostCodeProvider], [c].[Regex], [c].[SmsProvider], [c].[UpdatedAt], [c].[UpdatedBy]
FROM [Countries] AS [c]
WHERE [c].[IsDeleted] = 0
) AS [t0] ON [t].[OperationCountryId] = [t0].[Id]
WHERE ([w].[IsDeleted] = 0) AND ((((([w].[FirstName] LIKE #__filter_SearchTerm_0 + N''%'' AND (LEFT([w].[FirstName], LEN(#__filter_SearchTerm_0)) = #__filter_SearchTerm_0)) OR (#__filter_SearchTerm_0 = N'''')) OR (([w].[LastName] LIKE #__filter_SearchTerm_1 + N''%'' AND (LEFT([w].[LastName], LEN(#__filter_SearchTerm_1)) = #__filter_SearchTerm_1)) OR (#__filter_SearchTerm_1 = N''''))) OR (([w].[UserName] LIKE #__filter_SearchTerm_2 + N''%'' AND (LEFT([w].[UserName], LEN(#__filter_SearchTerm_2)) = #__filter_SearchTerm_2)) OR (#__filter_SearchTerm_2 = N''''))) OR EXISTS (
SELECT 1
FROM [Accounts] AS [x]
WHERE (([x].[IsDeleted] = 0) AND (([x].[AccountRef] LIKE #__filter_SearchTerm_3 + N''%'' AND (LEFT([x].[AccountRef], LEN(#__filter_SearchTerm_3)) = #__filter_SearchTerm_3)) OR (#__filter_SearchTerm_3 = N''''))) AND ([t].[Id] = [x].[AccountHolderId])))
ORDER BY [w].[LastActivity] DESC, [w].[Id], [t].[Id]',N'#__p_4 int,#__filter_SearchTerm_0 nvarchar(100),#__filter_SearchTerm_1 nvarchar(100),#__filter_SearchTerm_2 nvarchar(256),#__filter_SearchTerm_3 nvarchar(450)',#__p_4=10,#__filter_SearchTerm_0=N'james',#__filter_SearchTerm_1=N'james',#__filter_SearchTerm_2=N'james',#__filter_SearchTerm_3=N'james'
Finally this is my SQL query that returns whatever is necessary in less than 100 ms:
declare #searchTerm varchar(100) = '%james%'
select top 10
u.Id,
u.UserName,
u.FirstName,
u.LastName,
u.LastActivity,
u.CreatedAt,
a.Balance,
a.AccountRef,
ah.AccountHolderLevel,
u.Email,
r.Name
from Users u
join AccountHolders ah on ah.ObjectId = u.Id
join Accounts a on ah.Id = a.AccountHolderId
join UserRoles ur on ur.UserId = u.Id
join Roles r on r.Id = ur.RoleId
where FirstName like #searchTerm or LastName like #searchTerm or u.UserName like #searchTerm or FirstName + ' ' + LastName like #searchTerm or a.AccountRef like #searchTerm
and a.CurrencyId = ah.OperationCountryId
The columns I am searching are all indexed by the way, so that's not a problem. I know that the new EF-Core has many performance improvements. Unfortunately, I cannot update due to sheer number of breaking changes.
I am not sure splitting query into 2 (one for users and one for account) would work well, because there will be joins all over again. If I cannot find a solution using I plan converting my query to a view, but I want to do it as a last resort, since our convention is to use EF as much as possible. And I refuse to believe that EF does not have a solution. This is not actually a complex query at all and I am sure a fairly common use case.
So, what is the best way to optimize this query using EF-Core?
So, what is the best way to optimize this query using EF-Core?
Many things have changed in EF Core query pipeline since 2.1 (3.0, 3.1, 5.0 and now working on 6.0), but some general rules can be used, with the goal of getting rid of the client side query evaluation (which starting with 3.0 is not supported at all, so it's good to start preparing for the switch - support for 2.1 ends August this year).
The first would be to remove all these Include / ThenInclude. If the query is projecting the result in DTO without involving entity instances, then all these are redundant/not needed and removing them will ensure the query gets fully translated to SQL.
var query = _dbContext.Users.AsQueryable();
// Apply filters...
The next is the Roles collection. You must remove Mapper.Map call, otherwise it can't be translated. In general either use AutoMapper mappings and ProjectTo to fully handle the projection, or not use it at all (never put Map method calls inside query expression tree). According to your SQL, it should be something like this
Roles = user.UserRoles.Select(ur => ur.Role)
.Select(r => new RoleDTO { Name = r.Name })
.ToList(),
Actually EF Core will execute this as separate query (a behavior broken by "single query mode" in 3.x, and brought back optionally with 6.0 "split query mode"), so it is is important to have ToList() call at the end, otherwise you'll get N + 1 queries rather than 2.
Finally, the Single() call. It can be avoided by flattening the sub collection using correlated SelectMany, or its query syntax equivalent
from user in query
let ah = user.AccountHolder
from a in ah.Accounts
where a.CurrencyId == ah.OperationCountryId
The let statement is not mandatory, I've added it just for readability. Now you can use the range variables user, ah and a in the final select similar to table aliases in SQL.
Also since your SQL query doesn't really enforce single account match, there is no such enforcement in the LINQ query as well. If it was needed, then the equivalent of the Single can be achieved with SelectMany + Where + `Take(1), e.g.
from user in query
let ah = user.AccountHolder
from a in ah.Accounts
.Where(a => a.CurrencyId == ah.OperationCountryId)
.Take(1)
(a mixture of query and method syntax, but LINQ allows that)
So the final query would be something like this
from user in query
let ah = user.AccountHolder
from a in ah.Accounts
where a.CurrencyId == ah.OperationCountryId
select new //UserDTO
{
Id = user.Id,
FirstName = user.FirstName,
LastName = user.LastName,
Username = user.UserName,
Email = user.Email,
Roles = user.UserRoles.Select(ur => ur.Role)
.Select(r => new RoleDTO { Name = r.Name })
.ToList(),
LastActivity = user.LastActivity,
CreatedAt = user.CreatedAt,
EmailConfirmed = user.EmailConfirmed,
AccountBalance = a.Balance,
AccountReference = a.AccountRef
}
and should translate to very similar to the handcrafted SQL. And hopefully execute faster similar to it.

Need help in converting SQL query to LINQ

I am new to the world of LINQ and hence I am stuck at converting one sql query to LINQ.
My SQL query is:
select COUNT(DISTINCT PAYER) as count,
PPD_COL FROM BL_REV
where BL_NO_UID = 1084
GROUP BY PPD_COL
The desired output is:
Count PPD_COL
12 P
20 C
I have written something like below in LINQ:
var PayerCount = from a in LstBlRev where a.DelFlg == "N"
group a by new { a.PpdCol} into grouping
select new
{
Count = grouping.First().PayerCustCode.Distinct().Count(),
PPdCol = (grouping.Key.PpdCol == "P") ? "Prepaid" : "Collect"
};
But it is not giving me the desired output. The count is returned same for PPD_COL value P & C. What am I missing here?
Change the groupby as following. in the group group only the property you need and then in thr by no need to create an anonymous object - just the one property you are grouping by.
var PayerCount = from a in LstBlRev
where a.DelFlg == "N"
group a.PayerCustCode by a.PpdCol into grouping
select new
{
Count = grouping.Distinct().Count(),
PPdCol = grouping.Key == "P" ? "Prepaid" : "Collect"
};

Convert SQL to LINQ with group by

I'm stumped trying to convert the following sql to linq:
SELECT t.* FROM(SELECT mwfieldid,MAX([TimeStamp]) AS MaxValue, BatchDocumentID
FROM mw_BatchField
GROUP BY mwfieldid,BatchDocumentID) x
JOIN mw_BatchField t ON x.mwfieldid = t.mwfieldid
AND x.MaxValue = t.TimeStamp
and x.BatchDocumentID = t.BatchDocumentID
So far I had to convert it to a stored proc to get it to work. I'd rather know how to write this correctly in linq. I tried using a sql to linq converter (http://www.sqltolinq.com/) which produced this code that had errors in it: (Are these converters any good? It didn't seem to produce anything useful with a few tries.)
From x In (
(From mw_BatchFields In db.mw_BatchFields
Group mw_BatchFields By
mw_BatchFields.MWFieldID,
mw_BatchFields.BatchDocumentID
Into g = Group
Select
MWFieldID,
MaxValue = CType(g.Max(Function(p) p.TimeStamp),DateTime?),
BatchDocumentID)
)
Join t In db.mw_BatchFields
On New With { .MWFieldID = CInt(x.MWFieldID), .MaxValue = CDate(x.MaxValue), .BatchDocumentID = CInt(x.BatchDocumentID) }
Equals New With { .MWFieldID = t.MWFieldID, .MaxValue = t.TimeStamp, .BatchDocumentID = t.BatchDocumentID }
Select
BatchFieldID = t.BatchFieldID,
BatchDocumentID = t.BatchDocumentID,
MWFieldID = t.MWFieldID,
TimeStamp = t.TimeStamp,
value = t.value,
DictionaryValue = t.DictionaryValue,
AutoFilled = t.AutoFilled,
employeeID = t.employeeID
Seems like a lot of code for such a simple query, and it doesn't compile.
So for every combination of mwfieldid and BatchDocumentID you want all columns of the row with the highest TimeStamp? This is something which is much easier to express in LINQ than SQL so I'm not surprised that an automated converter is making a meal of it.
You should be able to do:
Mw_BatchFields.GroupBy(x => new { x.Mwfieldid, x.BatchDocumentId })
.SelectMany(x => x.Where(y => y.TimeStamp == x.Max(z => z.TimeStamp)))
This (like your SQL) will return multiple rows per grouping key if there is more than one row in the group that shares the same maximum TimeStamp. If you only want row per key, you could use:
Mw_BatchFields.GroupBy(x => new { x.Mwfieldid, x.BatchDocumentId })
.Select(x => x.OrderByDescending(y => y.TimeStamp).First())
Edit:
Sorry, just twigged that you're working in VB, not C#, so not quite what you were looking for, but if you can live with the lambda syntax style, I think the above can be translated as:
Mw_BatchFields.GroupBy(Function(x) New With {x.Mwfieldid, x.BatchDocumentId}).Select(Function(x) x.OrderByDescending(Function(y) y.TimeStamp).First())
and:
Mw_BatchFields.GroupBy(Function(x) New With {x.Mwfieldid, x.BatchDocumentId}).SelectMany(Function(x) x.Where(Function(y) y.TimeStamp = x.Max(Function(z) z.TimeStamp)))

how can I change a sql query to linq 2 nhibernate?

I have a query that is driving me crazy,,when I run it in sql it works fine but I dont know how to change it to linq to sql
the query is:
SELECT organizationstructure.PositionTitle.Title, organizationstructure.Person.FirstName, organizationstructure.Person.LastName,
organizationstructure.Department.Name
FROM organizationstructure.Department INNER JOIN
organizationstructure.Accountability AS Accountability_1 ON organizationstructure.Department.PartyId = Accountability_1.ParentPartyId INNER JOIN
organizationstructure.Accountability INNER JOIN
organizationstructure.Person ON organizationstructure.Accountability.ChildPartyId = organizationstructure.Person.PartyId INNER JOIN
organizationstructure.Position ON organizationstructure.Accountability.ParentPartyId = organizationstructure.Position.PartyId ON
Accountability_1.ChildPartyId = organizationstructure.Position.PartyId INNER JOIN
organizationstructure.PositionTitle ON organizationstructure.Position.PositionTitleId = organizationstructure.PositionTitle.PositionTitleId
and I think this is wrong but I changed it to:
query// query is iqueryable of position
.Join(Repository<Accountability>.Find(), p => p.Id, a => a.Child.Id,
(p, a) => new Tuple<string, string, int?>(((Department)a.Parent).Name, p.PositionTitle.Title, p.Id))
.Join(Repository<Accountability>.Find(), p => p.Item3, p => p.Parent.Id,
(p, d) => new Tuple<string, string, int?, string>(p.Item1, p.Item2, p.Item3, d.Child == null ? string.Empty : string.Format("{0}", ((Person)d.Child).FirstName) + " " + ((Person)d.Child).LastName))
whats wrong with it or ow can i change this query??
Generally having to do excessive numbers of explicit joins in your linq to nhibernate is a sign that you've not mapped your database to your domain appropriately. Without mappings between objects then you will just be reproducing SQL in your linq, which is a bit of a waste of time.
Your mappings should specify the relationships between the objects in your domain - for instance, a "Person" might have a reference to a "PositionTitle". If you use the mappings to setup relationships in this way then your query could end up looking something like this:
var results =
from
p in mySession.Query<Person>
select
new PersonalDetails
{
Title = p.PositionTitle.Title,
FirstName = p.FirstName,
LastName = p.LastName
DepartmentName = p.Party.Department.Name
};

Help understanding LINQ expression

Hi i'm try to debug some code but can't work out what it's doing.
// For reference, the following is roughly equivalent to:
// select p.*
// from CTBEquitiesFiles as o inner join CTBEquitiesDetailStaging as p
// on o.ID = p.CTBEquitiesFiles.ID
// where o.SEFileDate = fileDate and o.SEFileType = 'W'
mdovar depositRows = fileRepo.Query(o => o.SEFileDate == fileDate && o.SEFileType.Equals("D"))
.Join<SE.TradeDetailFile, SE.TradeDetailStaging, int, SE.TradeDetailStaging>(
detailRepo.Get(),
o => o.ID,
p => p.CTBEquitiesFiles.ID,
(o, p) => p);
I'm fairly sure it's not doing what I want. I want to get all the rows from CTBEquitiesFilesDetailsStaging where the SEFileID column is the same as the ID column in CTBEquitiesFiles.
Thank you
If you want to see what queries are being issued to the database, you can use the sql profiler, or set the DataContext.Log property.