Different performance from SQL Server query from Management Studio vs EF Core 5 - sql

I wrote a simple EF Core query that makes a select on a table using some where clause to filter data: start date and finish date between the actual date and a field (DescrizioneCommessa) containing a value.
var query = _ctx.Commessas
.Where(x => (x.DataInizioCommessa.HasValue && x.DataInizioCommessa <= DateTime.Now) || !x.DataInizioCommessa.HasValue)
.Where(x => (x.DataFineCommessa.HasValue && x.DataFineCommessa >= DateTime.Now) || !x.DataFineCommessa.HasValue)
.Where(x => x.DescrizioneCommessa.Contains(pattern))
.OrderBy(x => x.DescrizioneCommessa);
To get the raw SQL I just execute the statement:
var sql = facis.ToQueryString();
And the resultant query is:
DECLARE #__pattern_0 nvarchar(50) = N'COMUNE';
SELECT *
FROM [Commessa] AS [c]
WHERE (([c].[DataInizioCommessa] IS NOT NULL AND [c].[DataInizioCommessa] <= GETDATE()) OR [c].[DataInizioCommessa] IS NULL)
AND (([c].[DataFineCommessa] IS NOT NULL AND ([c].[DataFineCommessa] >= GETDATE())) OR [c].[DataFineCommessa] IS NULL)
AND ((#__pattern_0 LIKE N'') OR (CHARINDEX(#__pattern_0, [c].[DescrizioneCommessa]) > 0))
ORDER BY [c].[DescrizioneCommessa]
I notice that it takes very long to perform the query comparing to its hand-written version:
SELECT *
FROM Commessa
WHERE (DescrizioneCommessa LIKE '%COMUNE%')
AND (DataInizioCommessa <= GETDATE() OR DataInizioCommessa IS NULL)
AND (DataFineCommessa >= GETDATE() OR DataFineCommessa IS NULL);
EF Query takes even more than one minute to elaborate, while the normal one is immediate.
I verified that the problem is this part of where clause:
AND ((#__pattern_0 LIKE N'') OR (CHARINDEX(#__pattern_0, [c].[DescrizioneCommessa]) > 0))
If I substitute the above line with:
AND (DescrizioneCommessa LIKE '%COMUNE%')
the problem is resolved, the performance is optimal.
Why this line
.Where(x => x.DescrizioneCommessa.Contains(pattern))
creates this issue?

This is documented behaviour in EF.Core as discussed on SO: Entity framework EF.Functions.Like vs string.Contains
As a general proposition, LIKE expressions do NOT work well with query optimisers. In larger datasets this becomes a serious problem, even though they work just fine in much smaller unoptimized sets. The optimisation is heavily dependent on the pattern being matched and if it is a ranged lookup or not. In your case the pattern cannot make use indexes, EF is simply trying to convert it into an expression that might be indexable, in which case after running the expression enough the rest of the database insights engines would advise you to implement an appropriate index.
Read about some other discussions about parsing String.Contains() to SQL in git hub: https://github.com/dotnet/efcore/issues/474
When you explicitly want to use SQL LIKE, EF Core added EF.Functions.Like():
Like(DbFunctions, String, String)
Like-operator in Entity Framework Core 2.0
var query = _ctx.Commessas
.Where(x => (x.DataInizioCommessa.HasValue && x.DataInizioCommessa <= DateTime.Now) || !x.DataInizioCommessa.HasValue)
.Where(x => (x.DataFineCommessa.HasValue && x.DataFineCommessa >= DateTime.Now) || !x.DataFineCommessa.HasValue)
.Where(x => EF.Functions.Like(x.DescrizioneCommessa, pattern)
.OrderBy(x => x.DescrizioneCommessa);

Related

Get different results with the same query in T-SQL and LINQ (EF Core)

I'm trying to retrieve records from a database in Azure and for the checks I run T-SQL queries directly and then pass them to LINQ against the EF Core context, but I'm running into this problem.
select FechaOrientativa,id, Archivo, Estado, Estudiar, Descripcion
from Concursos
where FechaOrientativa>=CAST( GETDATE() AS Date ) and Estudiar='pt'
order by FechaOrientativa, Archivo, Estado
When I filter for the records with FechaOrientativa greater than or equal to Today, the Estudiar field is equal to 'pt', I get 2,296 records.
Now in Angular, I do http.Get to my Web API where I execute the following:
[HttpGet("sintratar")]
public async Task<ActionResult<IEnumerable<Concurso>>> GetConcursosSinTratar()
{
return await _context.Concursos.Where(c => c.Estudiar == "pt" && c.FechaOrientativa >= DateTime.Now).OrderBy(c => c.FechaOrientativa).ToListAsync();
}
And to my surprise, I receive only 2,151 records and I can't find an explanation.
Any idea, please?
Thanks.
Compare both SQL and EF LINQ queries, the difference is:
From
Query/Code
Result
SQL
CAST( GETDATE() AS Date )
Returns current Date without Time.
EF LINQ
DateTime.Now
Returns current Date with Time.
Hence the queried results are different.
(Example: Only queried records with the date-time field equal/after the query date-time).
From Date and time functions, you are looking for DateTime.Today.
DateTime.Today
CONVERT(date, GETDATE())
return await _context.Concursos
.Where(c => c.Estudiar == "pt" && c.FechaOrientativa >= DateTime.Today)
.OrderBy(c => c.FechaOrientativa)
.ToListAsync();
Or you can use the SQL query in EF Core with .FromSqlRaw().
return await _context.Concursos
.FromSqlRaw(#"select FechaOrientativa,id, Archivo, Estado, Estudiar, Descripcion
from Concursos
where FechaOrientativa>=CAST( GETDATE() AS Date ) and Estudiar='pt'
order by FechaOrientativa, Archivo, Estado")
.ToList();

Converting SQL Server query to Entiry Framework

I need to convert a SQL query to Entity Framework
Sort the values ​​of a field {'115-F-G', '10 -H-G ', '98 -T-R'} in ascending order.
SELECT * FROM ReportePedido
Where PedidoId =145
Order By TipoProducto,
CONVERT(INT, SUBSTRING(EnderecoEstoque, 0, CHARINDEX('-',EnderecoEstoque,1)))
Result: '10 -H-G ','98 -T-R','115-F-G'
It is pretty easy to write as a Linq query, ordering part doesn't seem to worth to be done on server side, it could easily be done on client side:
var query = ctx.ReportePedido
.Where(r => r.PedidoId == 145)
.AsEnumerable()
.OrderBy(r => r.TipoProducto),
.ThenBy(r => int.TryParse(r.Split('-')[0], out int i)?i:int.MaxValue)

SQL - Combine search queries instead of multiple DB trips

A search term comes from UI to search entities of a table. The order that these search results should show up in UI is like this:
First: exact match
Second: starts with that term
Third: contains a word of that term
Forth: ends with that term
Fifth: contains the term in any matter
So I first got the entities from DB:
result = entities.Where(e => e.Name.Contains(searchTerm)).ToList();
And then I rearranged them in memory:
var sortedEntities = result.Where(e => e.Name.ToLower() == searchTerm.ToLower())
.Union(result.Where(e => e.Name.StartsWith(searchTerm, StringComparison.OrdinalIgnoreCase)))
.Union(result.Where(e => e.Name.Contains($" {searchTerm} ")))
.Union(result.Where(e => e.Name.EndsWith(searchTerm, StringComparison.OrdinalIgnoreCase)))
.Union(result.Where(e => e.Name.Contains(searchTerm)));
It was working fine until I added paging. Now if an exact match is on page 2 (in data coming from DB) it won't show up first.
The only solution I can think of is to separate the requests (so 5 requests in this case) and keep track of page size manually. My question is that is there a way to tell DB to respect that order and get the sorted data in one DB trip?
It took me some time to realize that you use Union in an attempt to order data by "match strength": first the ones that match exactly, then the ones that match with different case, etc. When I see Unions with predicates my Pavlov-conditioned mind translates it into ORs. I had to switch from thinking fast to slow.
So the problem is that there is no predictable sorting. No doubt, the chained Union statements do produce a deterministic final sort order, but it's not necessarily the order of the Unions, because each Union also executes an implicit Distinct. The general rule is, if you want a specific sort order, use OrderBy methods.
Having said that, and taking...
var result = entities
.Where(e => e.Name.Contains(searchTerm))
.Skip((pageNumber - 1) * pageSize)
.Take(pageSize).ToList();
...the desired result seems to be obtainable by:
var sortedEntities = result
.OrderByDescending(e => e.Name == searchTerm)
.ThenByDescending(e => e.Name.ToLower() == searchTerm.ToLower())
.ThenByDescending(e => e.Name.StartsWith(searchTerm, StringComparison.OrdinalIgnoreCase))
... etc.
(descending, because false orders before true)
However, if there are more matches than pageSize the ordering will be too late. If pageSize = 20 and item 21 is the first exact match this item will not be on page 1. Which means: the ordering should be done before paging.
The first step would be to remove the .ToList() from the first statement. If you remove it, the first statement is an IQueryable expression and Entity Framework is able to combine the full statement into one SQL statement. The next step would be to move Skip/Take to the end of the full statement and it'll also be part of the SQL.
var result = entities.Where(e => e.Name.Contains(searchTerm));
var sortedEntities = result
.OrderByDescending(e => e.Name == searchTerm)
.ThenByDescending(e => e.Name.ToLower() == searchTerm.ToLower())
.ThenByDescending(e => e.Name.StartsWith(searchTerm, StringComparison.OrdinalIgnoreCase))
... etc
.Skip((pageNumber - 1) * pageSize)
.Take(pageSize).ToList();
But now a new problem blows in.
Since string comparison with StringComparison.OrdinalIgnoreCase isn't supported Entity Framework will auto-switch to client-side evaluation for part of the statement. All of the filtered results will be returned from the database, but most of the the ordering and all of the paging will be done in memory.
That may not be too bad when the filter is narrow, but very bad when it's wide. So, ultimately, to do this right, you have to remove StringComparison.OrdinalIgnoreCase and settle with a somewhat less refined match strength. Bringing us to the
End result:
var result = entities.Where(e => e.Name.Contains(searchTerm));
var sortedEntities = result
.OrderByDescending(e => e.Name == searchTerm)
.ThenByDescending(e => e.Name.StartsWith(searchTerm))
.ThenByDescending(e => e.Name.Contains($" {searchTerm} "))
.ThenByDescending(e => e.Name.EndsWith(searchTerm))
.ThenByDescending(e => e.Name.Contains(searchTerm))
.Skip((pageNumber - 1) * pageSize)
.Take(pageSize).ToList();
Why "less refined"? Because, according your comments, the database collation isn't case sensitive, so SQL can't distinguish exact matches by case without adding COLLATE statements. That's something we can't do with LINQ.

How to compare the month parts of two dates?

I am trying to query query the current month, here is my query:
$clients = $this->Clients;
$query = $clients->find();
if($this->Auth->user('role') !== 'admin'){
$query->where(['user_id =' => $this->Auth->user('id')]);
$query->where(['MONTH(dob) = ' => 'EXTRACT(month FROM (NOW()))']);
$query->order(['dob' => 'ASC']);
}
It returns 0 records (my field is a date type), however this query in phpmyadmin works:
SELECT * FROM `clients` WHERE MONTH(dob) = EXTRACT(month FROM (NOW()))
What am I doing wrong?
Just look at the actual generated query (check out your DBMS query log, or try DebugKit), it will look different, as the right hand side value in a key => value condition set is subject to parameter-binding/casting/quoting/escaping. In your case it will be treated as a string, so the condition will finally look something like:
WHERE MONTH(dob) = 'EXTRACT(month FROM (NOW()))'
That will of course not match anything.
You could pass the whole SQL snippet as a single array value, or as an expression object, that way it would be inserted into the query as is (do not insert user values that way, that would create an SQL injection vulnerability!), but I'd suggest to use portable function expressions instead.
CakePHP ships with functions expressions for EXTRACT and NOW, so you can simply do something like:
use Cake\Database\Expression\IdentifierExpression;
use Cake\Database\Expression\QueryExpression;
use Cake\ORM\Query;
// ...
$query->where(function (QueryExpression $exp, Query $query) {
return $exp->eq(
$query->func()->extract('MONTH', new IdentifierExpression('dob')),
$query->func()->extract('MONTH', $query->func()->now())
);
});
Looks a bit complicated, but it's worth it, it's cross DBMS portable as well as auto-quoting compatible. The generated SQL will look something like
WHERE EXTRACT(MONTH FROM (dob)) = (EXTRACT(MONTH FROM (NOW())))
See also
Cookbook > Database Access & ORM > Query Builder > Advanced Conditions
Cookbook > Database Access & ORM > Query Builder > Using SQL Functions
API > \Cake\Database\Expression\QueryExpression::eq()
API > \Cake\Database\FunctionsBuilder::extract()
API > \Cake\Database\FunctionsBuilder::now()

Nhibernate query condition within sum

I am trying the following code but nhibernate is throwing the following exception:
Expression type 'NhSumExpression' is not supported by this SelectClauseVisitor.
var data =
(
from a in session.Query<Activity>()
where a.Date.Date >= dateFrom.Date && a.Date.Date <= dateTo.Date
group a by new { Date = a.Date.Date, UserId = a.RegisteredUser.ExternalId } into grp
select new ActivityData()
{
UserID = grp.Key.UserId,
Date = grp.Key.Date,
Bet = grp.Sum(a => a.Amount < 0 ? (a.Amount * -1) : 0),
Won = grp.Sum(a => a.Amount > 0 ? (a.Amount) : 0)
}
).ToArray();
I've been looking around and found this answer
But I am not sure what I should use in place of the Projections.Constant being used in that example, and how I should create a group by clause consisting of multiple fields.
It looks like your grouping over multiple columns is correct.
This issue reported in the NHibernate bug tracker is similar: NH-2865 - "Expression type 'NhSumExpression' is not supported by this SelectClauseVisitor."
Problem is that apart from the less-than-helpful error message, it's not really a bug as such. What happens in NH-2865 is that the Sum expression contains something which NHibernate doesn't know how to convert into SQL, which result in this exception being thrown by a later part of the query processing.
So the question is, what does you sum expression contains that NHibernate cannot convert? The thing that jumps to mind is the use of the ternary operator. I believe the NHibernate LINQ provider has support for the ternary operator, but maybe there is something in this particular combination that is problematic.
However, I think your expressions can be written like this instead:
Bet = grp.Sum(a => Math.Min(a.Amount, 0) * -1), // Or Math.Abs() instead of multiplication.
Won = grp.Sum(a => Math.Max(a.Amount, 0))
If that doesn't work, try to use a real simple expression instead, like the following. If that works, we at least know the grouping itself work as expected.
Won = grp.Sum(a => a.Amount)