This question may have been answered somewhere (and if so I would love to have the link!). But since I don't know what type of linq query I'm trying to do, I haven't been able to find anything to help me.
I already have a very simple query setup to get all records of type DomainSequences.
var query = db.DomainSequences;
There are different types of domain sequences. I want to get all of them. However, for records that have a DomainType attribute equal to 'AT', then I only want to return some of those of 'AT' records (ie. do a where clause on those to filter the 'AT' records but still want to return all non 'AT' records.
More simply put, I want to get all DomainSequence records, but for the records that have a DomainType == "AT", then return those only if they meet certain conditions.
I can think of a way of doing it by doing something like:
query.Where(x => x.DomainType != "AT" || (x.DomainType == "AT" && AT conditions....));
I think this should work but the problem comes when I have to do subfilters on other columns and then it starts to get messier and complicated very quickly.
Ideally, I would like to do something like
query.Where(x => x.DomainType == "AT" || x.DomainType == "KS")
.WhereIf(y => y.DomainType == "AT" then apply filter on those)
.WhereIf(z => z.DomainType == "KS" then apply filter to these);
I'm not sure if there is a way to do this type of subfilter in LINQ or in SQL (though I imagine there is). Any suggestions on how this could be done relatively cleanly?
Thanks!
It's really not all that different than what you might write in SQL. In fact, if you would try the query expression syntax approach, you might find that particular connection easier to make.
var query = from domain in db.DomainSequences
where (domain.DomainType == "AT" && domain.Foo == 42 && ...)
|| (domain.DomainType == "KS" && domain.Foo = 117 && ...)
select domain;
That maps nicely to the SQL you might expect to write
SELECT *
FROM DomainSequences
WHERE
(DomainType = 'AT' AND Foo = 42 AND ...)
OR
(DomainType = 'KS' AND Foo = 117 AND ...)
You could, of course, keep it in the fluent extension method syntax using lambdas, of course, it's just a single .Where(...) invocation, after all.
query.Where(x => x.DomainType == "AT" || x.DomainType == "KS")
.Where(y => y.DomainType != "AT" || other filter)
.Where(z => z.DomainType != "KS" || other filter);
Related
I wrote a simple EF Core query that makes a select on a table using some where clause to filter data: start date and finish date between the actual date and a field (DescrizioneCommessa) containing a value.
var query = _ctx.Commessas
.Where(x => (x.DataInizioCommessa.HasValue && x.DataInizioCommessa <= DateTime.Now) || !x.DataInizioCommessa.HasValue)
.Where(x => (x.DataFineCommessa.HasValue && x.DataFineCommessa >= DateTime.Now) || !x.DataFineCommessa.HasValue)
.Where(x => x.DescrizioneCommessa.Contains(pattern))
.OrderBy(x => x.DescrizioneCommessa);
To get the raw SQL I just execute the statement:
var sql = facis.ToQueryString();
And the resultant query is:
DECLARE #__pattern_0 nvarchar(50) = N'COMUNE';
SELECT *
FROM [Commessa] AS [c]
WHERE (([c].[DataInizioCommessa] IS NOT NULL AND [c].[DataInizioCommessa] <= GETDATE()) OR [c].[DataInizioCommessa] IS NULL)
AND (([c].[DataFineCommessa] IS NOT NULL AND ([c].[DataFineCommessa] >= GETDATE())) OR [c].[DataFineCommessa] IS NULL)
AND ((#__pattern_0 LIKE N'') OR (CHARINDEX(#__pattern_0, [c].[DescrizioneCommessa]) > 0))
ORDER BY [c].[DescrizioneCommessa]
I notice that it takes very long to perform the query comparing to its hand-written version:
SELECT *
FROM Commessa
WHERE (DescrizioneCommessa LIKE '%COMUNE%')
AND (DataInizioCommessa <= GETDATE() OR DataInizioCommessa IS NULL)
AND (DataFineCommessa >= GETDATE() OR DataFineCommessa IS NULL);
EF Query takes even more than one minute to elaborate, while the normal one is immediate.
I verified that the problem is this part of where clause:
AND ((#__pattern_0 LIKE N'') OR (CHARINDEX(#__pattern_0, [c].[DescrizioneCommessa]) > 0))
If I substitute the above line with:
AND (DescrizioneCommessa LIKE '%COMUNE%')
the problem is resolved, the performance is optimal.
Why this line
.Where(x => x.DescrizioneCommessa.Contains(pattern))
creates this issue?
This is documented behaviour in EF.Core as discussed on SO: Entity framework EF.Functions.Like vs string.Contains
As a general proposition, LIKE expressions do NOT work well with query optimisers. In larger datasets this becomes a serious problem, even though they work just fine in much smaller unoptimized sets. The optimisation is heavily dependent on the pattern being matched and if it is a ranged lookup or not. In your case the pattern cannot make use indexes, EF is simply trying to convert it into an expression that might be indexable, in which case after running the expression enough the rest of the database insights engines would advise you to implement an appropriate index.
Read about some other discussions about parsing String.Contains() to SQL in git hub: https://github.com/dotnet/efcore/issues/474
When you explicitly want to use SQL LIKE, EF Core added EF.Functions.Like():
Like(DbFunctions, String, String)
Like-operator in Entity Framework Core 2.0
var query = _ctx.Commessas
.Where(x => (x.DataInizioCommessa.HasValue && x.DataInizioCommessa <= DateTime.Now) || !x.DataInizioCommessa.HasValue)
.Where(x => (x.DataFineCommessa.HasValue && x.DataFineCommessa >= DateTime.Now) || !x.DataFineCommessa.HasValue)
.Where(x => EF.Functions.Like(x.DescrizioneCommessa, pattern)
.OrderBy(x => x.DescrizioneCommessa);
We created a helper function to create wheres easier. It works fine with eq, neq, lt and gt. Now we're trying to add is null/is not null (for a date column, not sure if that matters).
The critical part of the function looks like this:
// This is ran in a loop for every attribute
const query = `${attribute}` ${comparator} :value${index}`;
// if the checked 'value' is NULL then use IsNull(), same for NOT NULL, otherwise simply use value
const params = { [`value${index}`]: value == 'NULL' ? IsNull() : value === 'NOT NULL' ? Not(IsNull()) : value};
// Add this (sub)query to the qb
qb.andWhere(query, params);
Now we get an error saying this:
You have an error in your SQL syntax; check the manual that
corresponds to your MySQL server version for the right syntax to use
near ’_type = ‘not’, _value = ‘[object Object]‘, _useParameter =
true, `_multipl’ at line 1"
Value is [object Object] - which kind of makes sense if we use IsNotNull(), right?
As far as I understand from this comment, IsNull() and Not(IsNull()) should work like we are trying to.
We use #nestjs/typeorm 7.1.5.
To check for NULL you need
qb.andWhere(`${attribute} IS NULL`)
To check for NOT NULL you need
qb.andWhere(`${attribute} IS NOT NULL`)
(Note: Omit the second argument, parameters, for these cases).
From your code seems you are using string values 'NULL' and 'NOT NULL' as the value arguments and checking these as special cases. Your code will now look like this:
if ((value == 'NULL' && comparator == '=') || (value == 'NOT NULL' && comparator == '<>'))
qb.andWhere(`${attribute} IS NULL`);
if ((value == 'NOT NULL' && comparator == '=') || (value == 'NULL' && comparator == '<>'))
qb.andWhere(`${attribute} IS NOT NULL`);
else
qb.andWhere(`${attribute} ${comparator} :value${index}`, { [`value${index}`]: value});
(In the code above I check for '=' and '<>' which are standard SQL comparison operators. If your SQL dialect uses 'eq' and 'ne' in place of '=' and '<>', which you mention in your question, you will need to change the code above. If so please update your question and add the appropriate tag to say which SQL database you are using).
When you test this, I recommend that you turn on TypeOrm full logging so you can see the actual generated SQL and you be able to quickly solve any problems. See TypeOrm logging.
so, I want to use conditional logic in my code, which the condition is when I got response.response_code == '00' so it will run
And match response == res_3[0]
And match each response.data.bills == res_3[1]
And if response.response_code != '00' , it will run
And match response == res_3
And match each response.data.bills == res_3
so, what is the best conditional logic for this case ??
Read the docs please: https://github.com/intuit/karate#conditional-logic
Use a second feature file:
* eval if (response.response_code != '00') karate.call('it-will-run.feature')
Note: you can't use match where JavaScript is expected, refer: https://stackoverflow.com/a/53961806/143475
A search term comes from UI to search entities of a table. The order that these search results should show up in UI is like this:
First: exact match
Second: starts with that term
Third: contains a word of that term
Forth: ends with that term
Fifth: contains the term in any matter
So I first got the entities from DB:
result = entities.Where(e => e.Name.Contains(searchTerm)).ToList();
And then I rearranged them in memory:
var sortedEntities = result.Where(e => e.Name.ToLower() == searchTerm.ToLower())
.Union(result.Where(e => e.Name.StartsWith(searchTerm, StringComparison.OrdinalIgnoreCase)))
.Union(result.Where(e => e.Name.Contains($" {searchTerm} ")))
.Union(result.Where(e => e.Name.EndsWith(searchTerm, StringComparison.OrdinalIgnoreCase)))
.Union(result.Where(e => e.Name.Contains(searchTerm)));
It was working fine until I added paging. Now if an exact match is on page 2 (in data coming from DB) it won't show up first.
The only solution I can think of is to separate the requests (so 5 requests in this case) and keep track of page size manually. My question is that is there a way to tell DB to respect that order and get the sorted data in one DB trip?
It took me some time to realize that you use Union in an attempt to order data by "match strength": first the ones that match exactly, then the ones that match with different case, etc. When I see Unions with predicates my Pavlov-conditioned mind translates it into ORs. I had to switch from thinking fast to slow.
So the problem is that there is no predictable sorting. No doubt, the chained Union statements do produce a deterministic final sort order, but it's not necessarily the order of the Unions, because each Union also executes an implicit Distinct. The general rule is, if you want a specific sort order, use OrderBy methods.
Having said that, and taking...
var result = entities
.Where(e => e.Name.Contains(searchTerm))
.Skip((pageNumber - 1) * pageSize)
.Take(pageSize).ToList();
...the desired result seems to be obtainable by:
var sortedEntities = result
.OrderByDescending(e => e.Name == searchTerm)
.ThenByDescending(e => e.Name.ToLower() == searchTerm.ToLower())
.ThenByDescending(e => e.Name.StartsWith(searchTerm, StringComparison.OrdinalIgnoreCase))
... etc.
(descending, because false orders before true)
However, if there are more matches than pageSize the ordering will be too late. If pageSize = 20 and item 21 is the first exact match this item will not be on page 1. Which means: the ordering should be done before paging.
The first step would be to remove the .ToList() from the first statement. If you remove it, the first statement is an IQueryable expression and Entity Framework is able to combine the full statement into one SQL statement. The next step would be to move Skip/Take to the end of the full statement and it'll also be part of the SQL.
var result = entities.Where(e => e.Name.Contains(searchTerm));
var sortedEntities = result
.OrderByDescending(e => e.Name == searchTerm)
.ThenByDescending(e => e.Name.ToLower() == searchTerm.ToLower())
.ThenByDescending(e => e.Name.StartsWith(searchTerm, StringComparison.OrdinalIgnoreCase))
... etc
.Skip((pageNumber - 1) * pageSize)
.Take(pageSize).ToList();
But now a new problem blows in.
Since string comparison with StringComparison.OrdinalIgnoreCase isn't supported Entity Framework will auto-switch to client-side evaluation for part of the statement. All of the filtered results will be returned from the database, but most of the the ordering and all of the paging will be done in memory.
That may not be too bad when the filter is narrow, but very bad when it's wide. So, ultimately, to do this right, you have to remove StringComparison.OrdinalIgnoreCase and settle with a somewhat less refined match strength. Bringing us to the
End result:
var result = entities.Where(e => e.Name.Contains(searchTerm));
var sortedEntities = result
.OrderByDescending(e => e.Name == searchTerm)
.ThenByDescending(e => e.Name.StartsWith(searchTerm))
.ThenByDescending(e => e.Name.Contains($" {searchTerm} "))
.ThenByDescending(e => e.Name.EndsWith(searchTerm))
.ThenByDescending(e => e.Name.Contains(searchTerm))
.Skip((pageNumber - 1) * pageSize)
.Take(pageSize).ToList();
Why "less refined"? Because, according your comments, the database collation isn't case sensitive, so SQL can't distinguish exact matches by case without adding COLLATE statements. That's something we can't do with LINQ.
I need some help in building the correct query. I have an Employees table. I need to get a list of all employees, that EENO (Employee ID) contains a string from a supplied array of partial Employee IDs.
When I use this code
// IEnumerable<string> employeeIds is a collection of partial Employee IDs
IQueryable<Employee> query = Employees;
foreach (string id in employeeIds)
{
query = query.Where(e => e.EENO.Contains(id));
}
return query;
I will get something like:
SELECT *
FROM Employees
WHERE EENO LIKE '%1111111%'
AND EENO LIKE '%2222222%'
AND EENO LIKE '%3333333%'
AND EENO LIKE '%4444444%'
Which doesn't make sense.
I need "OR" instead of "AND" in resulting SQL.
Thank you!
UPDATE
This code I wrote using PredicateBuilder works perfectly when I need to include these employees.
var predicate = PredicateBuilder.False<Employee>();
foreach (string id in employeeIds)
{
var temp = id;
predicate = predicate.Or(e => e.EENO.Contains(temp));
}
var query = Employees.Where(predicate);
Now, I need to write an opposite code, to exclude these employees,
here it is but it is not working: the generated SQL is totally weird.
var predicate = PredicateBuilder.False<Employee>();
foreach (string id in employeeIds)
{
var temp = id;
predicate = predicate.And(e => !e.EENO.Contains(temp)); // changed to "And" and "!"
}
var query = Employees.Where(predicate);
return query;
It's supposed to generate SQL Where clause like this one:
WHERE EENO NOT LIKE '%11111%'
AND NOT LIKE '%22222%'
AND NOT LIKE '%33333%'
But it's not happening
The SQL generated is this: http://i.imgur.com/9MDP7.png
Any help is appreciated. Thanks.
Instead of the foreach, just build the IQueryable once:
query = query.Where(e => employeeIds.Contains(e.EENO));
I'd take a look at http://www.albahari.com/nutshell/predicatebuilder.aspx. This has a great way of building Or queries, and is written by the guy that wrote LinqPad. The above link also has examples of usage.
I believe you can use Any():
var query = Employees.Where(emp => employeeIds.Any(id => id.Contains(emp.EENO)));
If you don't want to use a predicate builder, then the only other option is to UNION each of the collections together on an intermediate query:
// IEnumerable<string> employeeIds is a collection of partial Employee IDs
IQueryable<Employee> query = Enumerable.Empty<Employee>().AsQueryable();
foreach (string id in employeeIds)
{
string tempID = id;
query = query.Union(Employees.Where(e => e.EENO.Contains(tempID));
}
return query;
Also keep in mind that closure rules are going to break your predicate and only end up filtering on your last criteria. That's why I have the tempID variable inside the foreach loop.
EDIT: So here's the compendium of all the issues you've run across:
Generate ORs instead of ANDS
Done, using PredicateBuilder.
Only last predicate is being applied
Addressed by assigning a temp variable in your inner loop (due to closure rules)
Exclusion predicates not working
You need to start with the correct base case. When you use ORs, you need to make sure you start with the false case first, that way you only include records where AT LEAST ONE predicate matches (otherwise doesn't return anything). The reason for this is that the base case should just get ignored for purposes of evaluation. In other words false || predicate1 || predicate2 || ... really is just predicate1 || predicate2 || ... because you're looking for at least one true in your list of predicates (and you just need a base to build on). The opposite applies to the AND case. You start with true so that it gets "ignored" for purposes of evaluation, but you still need a base case. In other words: true && predicate1 && ... is the same as predicate1 && .... Hope that addresses your last issue.