I have an EFCore query that, for the same parameter values and same database state, sometimes throws a NullReferenceException. Any ideas how I can identify where the problem lies?
The query looks like this (yes it's a somewhat complicated db structure):
var lines = _context.Calcs.Where(x => x.PlanId == planId && x.IsDeleted == false)
.Include(x => x.Taggings)
.ThenInclude(x => x.CalcTag)
.Include(x => x.Destination)
.ThenInclude(x => x.FormatCode)
.Include(x => x.Destination)
.ThenInclude(x => x.Dimensions.Where(vtd => vtd.PlanId == planId))
.Include(x => x.CalcDimensions)
.ThenInclude(x => x.Dimension)
.Include(x => x.CLVT)
.ThenInclude(x => x.ValueType)
.Include(x => x.CLVT)
.ThenInclude(x => x.CLVTJoinFields)
.ThenInclude(x => x.Dimension)
.Include(x => x.CLVT)
.ThenInclude(x => x.CLVTTagFields)
.Include(x => x.CLVT)
.ThenInclude(clvt => clvt.PreCalc)
.Include(x => x.CLVTPreCalcs)
.ThenInclude(x => x.CLVTJoinFields)
.Include(x => x.CLVTPreCalcs)
.ThenInclude(x => x.CLVTTagFields)
.AsSplitQuery()
.ToList();
For exactly the same planId and database state, sometimes when this executes it results in a NullReferenceException and sometimes it doesn't. It looks like the call stack is always the same when the exception is thrown, which is helpful.
Exception is:
System.NullReferenceException
HResult=0x80004003
Message=Object reference not set to an instance of an object.
Source=Microsoft.EntityFrameworkCore.Relational
StackTrace:
at Microsoft.EntityFrameworkCore.Query.Internal.SplitQueryingEnumerable`1.Enumerator.Dispose()
at System.Collections.Generic.List`1..ctor(IEnumerable`1 collection)
at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source)
at MyClass.MyMethod(Int32 planId) in C:\code\server\MyProduct\MyClass.cs:line 94
Searching for other NullReferenceException cases from EFCore it seems like it can happen when there's a null value in the database when the EF model would expect it to be non-nullable. But in this case the same data is being pulled back and often there's no exception, so I'm guessing that's not the case here. What else could cause this?
My next guess is that in the code path that causes the problem something else has happened with the DbContext, e.g. it's cached some objects, causing a different behavior for this same query. But my _context.ChangeTracker.DebugView.ShortView is empty when this exception is thrown, suggesting it's not that. My context usually has QueryTrackingBehavior==NoTracking and AutoDetectChangesEnabled==false unless it's in the middle of an update, but I tested without that (i.e. default tracking behavior) and got the same results.
I've used SQL Server extended events to trace the SQL that's executed. I see about 5 queries being started and I think the exception happens before rpc_completed fires for them all. I haven't yet analysed this thoroughly enough to know if it always throws the exception at the same point with receiving the SQL server results. But given that the results of the queries are the same each time I'm not sure that analysing this further is going to help much.
Any suggestions what could be causing the exception?
I'm using EFCore 5.0.12 (I was on 5.0.5 but upgraded and same problem happens)
<PackageReference Include="Microsoft.EntityFrameworkCore.SqlServer"
Version="5.0.12" />
UPDATE:
Now I think that there's multiple threads using the same DbContext at once. They're all from the same request but there's some code a bit like this:
foreach (var x in y)
{
tasks.Add(MyAsyncMethodThatUsesDbContext(planId, otherParams));
}
var z = await Task.WhenAll(tasks);
It's a fair bit more convoluted so not as obvious, but I'll see if it's that.
The problem was caused by multiple threads using the same DbContext, which is a naughty thing to do. My DbContext wasn't shared by multiple requests but within a single background Hangfire job processing there was code that created a series of Tasks and then called WaitAll on them, something like this:
foreach (var x in y)
{
tasks.Add(MyAsyncMethodThatUsesDbContext(planId, otherParams));
}
var z = await Task.WhenAll(tasks);
I thought there was something in EFCore that warned when you use the same DbContext from multiple threads, but perhaps it doesn't work either when running from a Hangfire job (i.e. no active asp request) or when the tasks are created all from the same thread?
One solution is to not parallelize the database access. Or alternatively ensure a separate DbContext is used for each task.
Related
I'm using EF Core 5. I want to make a query and use ProjectTo for mapping my result. But I need to make a filter in nested collection like this:
var questions = await _context
.MedicalDatas
.Include(p => p.MedicalDataItems)
.Include(p => p.MedicalAnswers.Where(x => x.AppUserId == userId))
.ProjectTo<MedicalDataDto>(_cp)
.ToListAsync();
The 2 navigation properties are included in destination type but when I check the SQL generated by EF Core, there is no where clause and it selects all medical answers from the database.
BUT without projecting, it works as expected.
What is wrong here?
you don't need include with projectto
var questions = await _context
.MedicalDatas
.Where(p => p.MedicalAnswers.Any(x => x.AppUserId == userId))
.ProjectTo<MedicalDataDto>(_cp)
.ToListAsync();
I'd like to use a couple of regexes in an index. This is a simplified example:
Map =
books =>
books.Select(x => new {Sentences = Regex.Split(x.Description, "<br>")})
.Select(x => new {Results = x.Sentences.Where(y => Regex.IsMatch(y, "foo"))})
.Where(x => x.Results.Any())
.Select(x => new {});
However, the following exception is thrown:
An exception of type 'Microsoft.CSharp.RuntimeBinder.RuntimeBinderException' occurred in System.Core.dll but was not handled in user code
Additional information: 'System.Array' does not contain a definition for 'Where'
I've tried .AsEnumerable() but to no avail. Can this be done?
To fix this, you need to call .AsEnumerable() on the thing you're .Select'ing off.
So instead of Regex.Split(...).Select(...), use Regex.Split(...).AsEnumerable().Select(...)
RavenDB relies on an internal class that knows how to map common extension methods.
Sometimes, however, it may not be able to dynamically resolve the extension methods. To help dynamic resolution, call the .AsEnumerable(), then the extension methods will be found.
Your code should look something like this:
Map =
books =>
books.Select(x => new {Sentences = Regex.Split(x.Description, "<br>")})
.AsEnumerable()
.Select(x => new {Results = x.Sentences.Where(y => Regex.IsMatch(y, "foo"))})
.Where(x => x.Results.Any())
.Select(x => new {});
I have used too much time to find a nice way to eager load child collections. so far this is what I got. It's working but I find it hard to believe that this is the best way to write this query
[Fact]
public void EagerLoadQueryOverWithFutureTest()
{
const long journalNr = 1470;
var query = _repo.QueryOver().Where(s => s.JournalNr == journalNr).Future<Sag>();
var sagsansoegning = _repo.QueryOver().Where(an => an.JournalNr == journalNr).Fetch(x => x.Sagsansoegning).Eager.Future<Sag>();
var noter = _repo.QueryOver().Where(n => n.JournalNr == journalNr).Fetch(x => x.Noter).Eager.Future<Sag>();
var filer = _repo.QueryOver().Where(f => f.JournalNr == journalNr).Fetch(x => x.Filer).Eager.Future<Sag>();
var emails = _repo.QueryOver().Where(e => e.JournalNr == journalNr).Fetch(x => x.Emails).Eager.Future<Sag>();
var journal = _repo.QueryOver().Where(j => j.JournalNr == journalNr).Fetch(x => x.Journal).Eager.Future<Sag>();
var sagsTilstand = _repo.QueryOver().Where(t => t.JournalNr == journalNr).Fetch(x => x.Tilstand).Eager.Future<Sag>();
var boligsocialEvalueringer = _repo.QueryOver().Where(b => b.JournalNr == journalNr).Fetch(x => x.BoligsocialEvaluering).Eager.Future<Sag>();
var result = query.SingleOrDefault();
result.JournalNr.Should().Be(journalNr);
result.Emails.Should().HaveCount(c => c > 20);
result.Filer.Should().HaveCount(c => c > 20);
}
I would like to append different point of view, not the answer how to use "future" to make only one round-trip to DB.
Not sure why you need to fetch all the properties. I guess it could be 1) to be used by your code, e.g. to render UI or 2) as the API which somehow serializes that and passes it to the client.
In the first scenario, I would suggest to use the session per request and use the lazy load. In the second, I would suggest to introduce one step: building the DTO before passing it to client. That could be done while session is still open, therefore use the lazy load.
Lazy load in both scenarios. So, in both cases I would suggest to use different approach then explicit fetching. It will use more or less the same amount of selects and round-trips to DB. But at the end it could be more effective. Because all that job would be left on the NHibernate
The thing is to change the eager-fetch (manual) loading with the batch-size (native) loading. See more here 19.1.5. Using batch fetching. In this case, NHibernate will process your query, and then will go again with few more queries to load the remaining data. But only if really needed
The advantage is, that you are not a prisoner of your queries. You can consume any property which could/should be available on the entity, until the session is open (I.e. during View rendering or serializing). You do not have to go and append another explicit query over join to get eager data.
So, while Future could be solution to the world were your connectivity to DB from APP is very slow... it is not the world were to use NHibernate (ORM). The advantage of the lazy mapping and ad hoc loading (with power of batching) is from maintenance perspective the right way... I'd say
I'm trying to wire NCommon and NH to multiple databases via the guidance at http://codeinsanity.com (see 'Configuring NCommon for multiple database support') and it works via the fluent approach suggested, below:
var configuration = NCommon.Configure.Using(adapter).ConfigureState<DefaultStateConfiguration>();
configuration.ConfigureData<NHConfiguration>(config => config.WithSessionFactory(() => _sessionFactories[0]).WithSessionFactory(() => _sessionFactories[1]));
This works as expected but as you can see the sessionFactories are hardcoded. What I'd really like to do is something like this:
foreach(ISessionFactory sessionFactory in _sessionFactories)
{
configuration.ConfigureData<NHConfiguration>(config => config.WithSessionFactory(() => sessionFactory));
}
But this throws the following exception:
Component NCommon.Data.NHibernate.NHUnitOfWorkFactory could not be registered. There is already a component with that name. Did you want to modify the existing component instead? If not, make sure you specify a unique name.
My hope is there's a proper way to wire-up n-SessionFactories without hardcoding them - but I'm just not seeing a solution. Any advice?
I thought I got this to work by delegating the SessionFactory piece to a method, as below:
configuration.ConfigureData<NHConfiguration>(config => ConfigureSessionFactories(config));
private void ConfigureSessionFactories(NHConfiguration configuration)
{
foreach (ISessionFactory sessionFactory in _sessionFactories)
{
configuration.WithSessionFactory(() => sessionFactory);
}
}
However, that only appears to be a solution. Say you have multiple session factories, you can use all of them to query, but only the last-added session factory will hit the database and return results. Queries against the other session factories will never hit the actual database and will return zero entities.
The only way I've been able to get this to work is:
configuration.ConfigureData<NHConfiguration>(config => config.WithSessionFactory(() => _sessionFactories[0]).WithSessionFactory(() => _sessionFactories[1]));
I'd rather not hard-code that though.. I'd much rather iterate over a loop of n-session factories... does anyone have an idea of how to accomplish this?
This NHibernate blog entry notes how detached QueryOver queries (analogous to DetachedCriteria) can be created (using QueryOver.Of<T>()). However, looking this over, it doesn't look analogous to me at all.
With DetachedCriteria, I would create my instance and set it up however I need, and afterwards call GetExecutableCriteria() to then assign the session and execute the query. With the "detached" QueryOver, most of the API is unavailable (ie, to add restrictions, joins, ordering, etc...) until I call GetExecutableQueryOver, which requires takes an ISession or IStatelessSession, at which point you are no longer disconnected.
How do you work with detached QueryOver instances?
EDIT:
Actual problem was related to how I'm storing the detached QueryOver instance:
public class CriteriaQuery<T>
{
internal protected QueryOver<T> _QueryOver { get; set; }
public CriteriaQuery()
{
_QueryOver = QueryOver.Of<T>();
}
// Snip
}
It should be a QueryOver<T, T>.
I'm using NHibernate 3.1.0.4000. The following code compiles successfully:
Employee salesRepAlias = null;
var query = QueryOver.Of<Customer>()
.JoinAlias(x => x.SalesRep, () => salesRepAlias)
.Where(x => x.LastName == "Smith")
.Where(() => salesRepAlias.Office.Id == 23)
.OrderBy(x => x.LastName).Asc
.ThenBy(x => x.FirstName).Asc;
return query.GetExecutableQueryOver(session)
.List();
This illustrates using restrictions, joins, and ordering on a detached QueryOver just like you would with a regular one.
Could you please post the code that demonstrates the API features that are unavailable?