Nhibernate queries fail itermitently - nhibernate

Question

I'm having a really hard time nailing down a problem coming from NHibernate (3.3). The application serves some 150 req/s in a six node NLB cluster. The application behaves ok generally, but sometimes maybe 1-2 day the following error comes in the logs and all queries fail.
I'm using adonet.batch_size of 15 and MultipleActiveResultSets=True is set to true. All controllers inherit from AsyncController (this is an ASP.NET MVC 4 app), so session state is read only to maximize concurrency.
Initially, we had a lot of trouble NHibernate Session handling because the AbstractBatcher did a pretty poor job managing which readers were related to which connection. So, in a high load, readers would try to read from connections that were already closed. We solved this by defining manually the lifetime of a SQLConnection which stabilized things.
However, the following comes up often. I think it is a symptom of something really wrong with the session factory. At this point, I'm thinking about reinitializing the session factory when an unhandled exception of this type is thrown, but that's not nice. Does anyone have any insight to why this happens?
System.NotSupportedException: PartialEvaluationExceptionExpression
at NHibernate.Linq.Visitors.HqlGeneratorExpressionTreeVisitor.VisitExpression(Expression expression)
at NHibernate.Linq.Visitors.HqlGeneratorExpressionTreeVisitor.VisitBinaryExpression(BinaryExpression expression)
at NHibernate.Linq.Visitors.HqlGeneratorExpressionTreeVisitor.VisitExpression(Expression expression)
at NHibernate.Linq.Visitors.HqlGeneratorExpressionTreeVisitor.VisitBinaryExpression(BinaryExpression expression)
at NHibernate.Linq.Visitors.HqlGeneratorExpressionTreeVisitor.VisitExpression(Expression expression)
at NHibernate.Linq.Visitors.HqlGeneratorExpressionTreeVisitor.VisitBinaryExpression(BinaryExpression expression)
at NHibernate.Linq.Visitors.HqlGeneratorExpressionTreeVisitor.VisitExpression(Expression expression)
at NHibernate.Linq.Visitors.QueryModelVisitor.VisitWhereClause(WhereClause whereClause, QueryModel queryModel, Int32 index)
at Remotion.Linq.Clauses.WhereClause.Accept(IQueryModelVisitor visitor, QueryModel queryModel, Int32 index)
at Remotion.Linq.QueryModelVisitorBase.VisitBodyClauses(ObservableCollection`1 bodyClauses, QueryModel queryModel)
at Remotion.Linq.QueryModelVisitorBase.VisitQueryModel(QueryModel queryModel)
at NHibernate.Linq.Visitors.QueryModelVisitor.GenerateHqlQuery(QueryModel queryModel, VisitorParameters parameters, Boolean root)
at NHibernate.Linq.NhLinqExpression.Translate(ISessionFactoryImplementor sessionFactory)
at NHibernate.Hql.Ast.ANTLR.ASTQueryTranslatorFactory.CreateQueryTranslators(String queryIdentifier, IQueryExpression queryExpression, String collectionRole, Boolean shallow, IDictionary`2 filters, ISessionFactoryImplementor factory)
at NHibernate.Engine.Query.QueryPlanCache.GetHQLQueryPlan(IQueryExpression queryExpression, Boolean shallow, IDictionary`2 enabledFilters)
at NHibernate.Impl.AbstractSessionImpl.GetHQLQueryPlan(IQueryExpression queryExpression, Boolean shallow)
at NHibernate.Impl.AbstractSessionImpl.CreateQuery(IQueryExpression queryExpression)
at NHibernate.Linq.DefaultQueryProvider.PrepareQuery(Expression expression, IQuery& query, NhLinqExpression& nhQuery)
at NHibernate.Linq.DefaultQueryProvider.Execute(Expression expression)
at NHibernate.Linq.DefaultQueryProvider.Execute[TResult](Expression expression)
at System.Linq.Queryable.FirstOrDefault[TSource](IQueryable`1 source)

Answer 1

For those who may be facing this issue, I thought I'd give some insight of two years in production (thousands of users) of a fix that we put in place to address this issue.
In the base constructor we have something like this:
public MvcBaseController()
{
try
{
var currentRepositorySession = RepositorySession.Current;
dbConnection = new SqlConnection(currentRepositorySession.GetConnectionString());
dbConnection.Open();
session = currentRepositorySession.Create(false, dbConnection);
}
catch (Exception e)
{
LogDebug("Error was thrown: " + e.Message);
}
}
Binding the db connection to the NHibernate session forcefully is the only way to ensure that the AbstractBatcher doesn't close the session before all readers are done. On Dispose we have the following:
protected override void Dispose(bool disposing)
{
if (disposing)
{
try
{
if (session != null)
{
session.Dispose();
session = null;
}
}
catch (Exception ex)
{
LogDebug("Error was thrown: " + ex.Message);
}
try
{
if (dbConnection != null)
{
dbConnection.Close();
dbConnection.Dispose();
dbConnection = null;
}
}
catch (Exception ex)
{
LogDebug("Error was thrown: " + ex.Message);
}
}
base.Dispose(disposing);
}
This means that a connection is opened for every requests, which is fine. We had to increase SQL Server workers, but the system has been stable for two years now. The only time we've seen NHibernate session factory break is when there is a lot of data paged from the database, say 1 million records. Then the application pool overflows, and is orphaned. These were mostly code bugs and not related to the concurrency issues described in the question.