I have to match records in SQL around a zip code with a min/max range
the challenge is that the data qualities are bad, some zipcodes are not numbers only
so I try to match "good zip codes" either by discarding bad ones or even keeping only digits
I dont know how to use Regex.Replace(..., #"[^\d]", "") instead of Regex.Match(..., #"\d") to fit in the query bellow
I get an error with the code bellow at runtime
I tried
Regex.IsMatch
SqlFunctions.IsNumeric
they all cause errors at runtime, here is the code :
var data = context.Leads.AsQueryable();
data = data.Include(p => p.Company).Include(p => p.Contact);
data = data.Where(p => Regex.IsMatch(p.Company.ZipCode, #"\d"));
data = data.Where(p => Convert.ToInt32(p.Company.ZipCode) >= range.Min);
data = data.Where(p => Convert.ToInt32(p.Company.ZipCode) <= range.Max);
here is the error :
System.InvalidOperationException: The LINQ expression 'DbSet<Lead>
.Join(
outer: DbSet<Company>,
inner: l => EF.Property<Nullable<int>>(l, "CompanyId"),
outerKeySelector: c => EF.Property<Nullable<int>>(c, "Id"),
innerKeySelector: (o, i) => new TransparentIdentifier<Lead, Company>(
Outer = o,
Inner = i
))
.Where(l => !(Regex.IsMatch(
input: l.Inner.ZipCode,
pattern: "\d")))' could not be translated. Either rewrite the query in a form that can be translated, or switch to client evaluation explicitly by inserting a call to either AsEnumerable(), AsAsyncEnumerable(), ToList(), or ToListAsync().
I am not sure how to solve this. I really don't see how AsEnumerable(), AsAsyncEnumerable(), ToList(), or ToListAsync() could help here
what do I do wrong ?
thanks for your help
Wnen you use querable list, Ef Core 5 is always trying to translate query to SQl, so you have to use code that SQL server could understand. If you want to use C# function you will have to download data to Server using ToList() or ToArray() at first and after this you can use any c# functions using downloaded data.
You can try something like this:
var data = context.Leads
.Include(p => p.Company)
.Include(p => p.Contact)
.Where(p =>
p.Company.Zipcode.All(char.IsDigit)
&& (Convert.ToInt32(p.Company.ZipCode) >= range.Min) //or >=1
&& ( Convert.ToInt32(p.Company.ZipCode) <= range.Max) ) // or <=99999
.ToArray();
I tried everything imaginable
all sorts of linq/ef trickeries, I even tried to define a DBFunction that was never found
once I had a running stored procedure written dirrectly in SQL, I ended up with a list, not with an IQueryable, so I was back to #1
finaly, I just created a new field in my table :
ZipCodeNum
which holds a filtered , converted version of the zipcode string
Related
I develop a web app with C# and I want to display these SQL queries in Swagger. So I want to convert them to Linq.
First SQL query:
SELECT TOP 3 HastalikIsmi, COUNT(*)
FROM Hastaliklar
GROUP BY HastalikIsmi
ORDER BY COUNT(*) DESC
Second SQL query:
SELECT TcNo, Isim, Soyisim, Hastaliklar.HastalikIsmi
FROM Calisanlar, Hastaliklar
WHERE Calisanlar.CalisanId = Hastaliklar.CalisanId
AND HastalikIsmi IN (SELECT TOP 3 HastalikIsmi
FROM Hastaliklar
GROUP BY HastalikIsmi
ORDER BY COUNT(*) DESC)
I barely work with raw SQL since I use EF most of the time (so I might have misinterpreted some of the commands) and neither did I have the time to test the second query so I appologize in advance if there are any mistakes.
First SQL query translated:
var result = hastaliklars.GroupBy(hastaliklar => hastaliklar.HastalikIsmi)
.Select(hastaliklarGroup =>
(HastalikIsmi: hastaliklarGroup.Key, Count: hastaliklarGroup.Count()))
.OrderByDescending(hastalikIsmiWithCount => hastalikIsmiWithCount.Count)
.Take(3)
.ToList();
Second SQL query translated:
var result2 = calisanlars.Join(hastaliklars,
calisanlar => calisanlar.CalisanId,
hastaliklar => hastaliklar.CalisanId,
(calisanlar, hastaliklar) =>
(Calisanlar: calisanlar, Hastaliklar: hastaliklar))
.Where(calisanlarAndHastaliklar => hastaliklars
.GroupBy(hastaliklar => hastaliklar.HastalikIsmi)
.OrderByDescending(hastaliklarGroup => hastaliklarGroup.Count())
.Take(3)
.Any(hastaliklarGroup =>
calisanlarAndHastaliklar.Hastaliklar.HastalikIsmi == hastaliklarGroup.Key))
.ToList();
Note that if you're working with an IQueryable<T> instead of an IEnumerable<T> you might get a runtime exception stating that the LINQ expression could not be translated and will be evaluated locally. The reason why this might happen is because an IQueryable<T> is not run locally on C# but rather just contains instructions on what you're trying to do and translates that into an SQL query string. Therefore, specific code can not be translated into SQL which causes this exception.
The easiest way to fix this would be to run the method AsEnumerable() before the method which can not be translated to SQL which forces only all of the previous instructions to be run via SQL and then reads and works with the results locally from there on out. However, it does come at a performance cost, so that it would be slower than running the entire query via SQL. Therefore the optimal solution would be to adjust the code so that all of it could be translated into SQL if possible.
I don't fully remember what code IQueryable<T> supports and what not but as a good start; Here's a link to the microsoft docs that lists all of the LinQ functions it supports: https://learn.microsoft.com/en-us/dotnet/framework/data/adonet/ef/language-reference/supported-and-unsupported-linq-methods-linq-to-entities?redirectedfrom=MSDN
NHibernate Version: 3.4.0.4000
I'm currently working on optimizing our code so that we can reduce the number of round trips to the database and am looking at a for loop that is one of the culprits. I'm having a hard time figuring out how to batch all of these iterations into a future that gets executed once when sent to SQL Server. Essentially each iteration of the loop causes 2 queries to hit the database!
foreach (var choice in lineItem.LineItemChoices)
{
choice.OptionVersion = _session.Query<OptionVersion>().Where(x => x.Option.Id == choice.OptionId).OrderByDescending(x => x.OptionVersionNumber).FirstOrDefault();
choice.ChoiceVersion = _session.Query<ChoiceVersion>().OrderByDescending(x => x.ChoiceVersionIdentity.ChoiceVersionNumber).Where(x => x.Choice.Id == choice.ChoiceId).FirstOrDefault();
}
One option is to extract OptionId and ChoiceId from all the LineItemChoices into two lists in local memory. Then issue just two queries, one for options and one for choices, giving these lists in .Where(x => optionIds.Contains(x.Option.Id)). This corresponds to SQL IN operator. This requires some postprocessing. You will get two result lists (transform to dictionary or lookup if you expect many results), that you need to process to populate the choice objects. This postprocessing is local and tends to be very cheap compared to database roundtrips. This option can be a bit tricky if the existing FirstOrDefault part is absolutely necessary. Do you expect there to be more than result for a single optionId? If not, this code could instead have used SingleOrDefault, which could just be dropped if converting to use IN-queries.
The other option is to use futures (https://nhibernate.info/doc/nhibernate-reference/performance.html#performance-future). For Linq it means to use ToFuture or ToFutureValue at the end, which also conflicts with FirstOrDefault I believe. The important thing is that you need to loop over all line item choices to initialize ALL queries BEFORE you access the value of any of them. So this is likely to also result in some postprocessing, where you would first store the future values in some list, and then in a second loop access the real value from each query to populate the line item choice.
If you to expect that the queries can yield more than one result (before applying FirstOrDefault), I think you can just use Take(1) instead, as that will still return an IQueryable where you can apply the future method.
The first option is probably the most efficient, since it will just be two queries and allow the database engine to make just one pass over the tables.
Keep the limit on the maximum number of parameters that can be given in an SQL query in mind. If there can be thousands of line item choices, you may need to split them in batches and query for at most 2000 identifiers per round trip.
Adding on the Oskar answer, NHibernate Futures was implement in NHibernate 2.1. It is available on method Future for collections and FutureValue for single values.
In your case, you could separate the IDs of the list in memory ...
var optionIds = lineItem.LineItemChoices.Select(x => x.OptionId);
var choiceIds = lineItem.LineItemChoices.Select(x => x.ChoiceId);
... and execute two queries using Future<T> to get two lits in one hit over the database.
var optionVersions = _session.Query<OptionVersion>()
.Where(x => optionIds.Contains(x.Option.Id))
.OrderByDescending(x => x.OptionVersionNumber)
.Future<OptionVersion>();
var choiceVersions = _session.Query<ChoiceVersion>()
.Where(x => choiceIds.Contains(x.Choice.Id))
.OrderByDescending(x => x.ChoiceVersionIdentity.ChoiceVersionNumber)
.Future<ChoiceVersion>();
After with all you need in memory, you could loop on the original collection you have and search in memory the data to fill up the choice object.
foreach (var choice in lineItem.LineItemChoices)
{
choice.OptionVersion = optionVersions.OrderByDescending(x => x.OptionVersionNumber).FirstOrDefault(x => x.Option.Id == choice.OptionId);
choice.ChoiceVersion = choiceVersions.OrderByDescending(x => x.ChoiceVersionIdentity.ChoiceVersionNumber).FirstOrDefault(x => x.Choice.Id == choice.ChoiceId);
}
I have a WCF layer to a quite complex database that i read using LINQ to Entities. Recently though, a weird error has emerged. I get null references when executing the queries, but only sometimes (the data in the database is NOT changing). I have two general cases described below, and it's EITHER the one OR the other failing. If i restart the site accessing the service so that the connection is reinitialized, the roles switch. Now the other case is failing, while the first works. So.. I'm quite confused. Does anybody come to think of anything?
Case 1:
string code = "xxx";
int version = 1;
Language lang = Language.en;
var languages = this.uc.MainTable
.Expand(a => a.Program.Area)
.Expand(a => a.Plang)
.Expand(a => a.Lang)
.Where(a =>
a.Program.Code.Equals(code, StringComparison.OrdinalIgnoreCase)
&& a.Lang != null
&& a.Program.Version == version)
.ToList();
var language = languages.Where(a => a.Lang.LangID == (int)lang).SingleOrDefault();
In the case above, when enumerating language, a.Lang is null, and the method fails. Again, this is only sometimes. With the exact same input parameters and database content, the query works again after reinitializing the service connection. The second case seems to fail when the first one successes and vice versa. The other case is this:
Case 2:
Language lang = Language.en;
var programs = this.uc.SomeTable
.Expand(a => a.Program)
.Expand(a => a.Plang)
.Expand(a => a.Program.Level)
.Where(a =>
a.Program.Version == 1
&& a.LangID == (int)lang
&& a.Program.Occasion.Any(b => b.StatusID != 3));
programs = programs.Where(a => a.Program.Occasion.Any(b => b.Date == "2010-01-01"));
When programs is enumerated, a.Program.Occasion in the last line is null and causes a null reference. Again, this is intermittent, despite identical database and parameters.
I'm grateful for even the slightest idea of what might be causing this...
Answered here: http://social.msdn.microsoft.com/Forums/en-US/adodotnetdataservices/thread/9fb8648a-fb4f-4ebd-ad5a-c911cd00a812.
This is usually caused by DataServiceContext.MergeOption. The default setting means that client will never update the entity after the first query. Which means that additional queries will not populate navigation properties even if they were expanded in the query. The fix is to set MergeOption to OverwriteChanges.
I simply want to include a row number against the returned results of my query.
I found the following post that describes what I am trying to achieve but gives me an exception
http://vaultofthoughts.net/LINQRowNumberColumn.aspx
"An expression tree may not contain an assignment operator"
In MS SQL I would just use the ROWNUMBER() function, I'm simply looking for the equivalent in LINQ.
Use AsEnumerable() to evaluate the final part of your query on the client, and in that final part add a counter column:
int rowNo = 0;
var results = (from data in db.Data
// Add any processing to be performed server side
select data)
.AsEnumerable()
.Select(d => new { Data = d, Count = ++rowNo });
I'm not sure whether LINQ to SQL supports it (but it propably will), but there's an overload to the Queryable.Select method that accepts an lambda with an indexer. You can write your query as follows:
db.Authors.Select((author, index) => new
{
Lp = index, Name = author.Name
});
UPDATE:
I ran a few tests, but unfortunately LINQ to SQL does not support this overload (both 3.5sp1 and 4.0). It throws a NotSupportedException with the message:
Unsupported overload used for query
operator 'Select'.
LINQ to SQL allows you to map a SQL function. While I've not tested this, I think this construct will work:
public partial class YourDataContext : DatContext
{
[Function(Name = "ROWNUMBER")]
public int RowNumber()
{
throw InvalidOperationException("Not called directly.");
}
}
And write a query as follows:
from author in db.Authors
select new { Lp = db.RowNumber(), Name = author.Name };
"Method 'Boolean Contains(System.String)' has no supported translation to SQL."
query is IsQueryable but this stopped working:
foreach (string s in collection1)
{
if (s.Length > 0)
{
query = query.Where(m => m.collection2.Contains(s));
}
}
UPDATE: it works when i make query "ienumerable" instead of iqueryable. What would be the way to get same result using linq instead of iterating through loop?
Try this:
query = query.Where(m => m.collection2.ToList().Contains(s));
^^^^^^^^
Take a look at this answer from stackoverflow.
It looks like the resulting query would need access to something that the database
has no way of reaching, because the info is in memory.
Since m.collection2 is in the database, don't use Contains. Use Any
m.collection2.Any(x => x == s)
It looks like the error you are seeing is coming from the collection collection 2. Have you tried wrappering the m.collection2 in another function which returns true or false? Is this LINQ syntax?