Strange NHibernate errors - nhibernate

Question

I'm seeing some strange NHibernate errors that occur sporadically in our production environment, but not on our test servers. Days will go by with no errors, then we'll receive an error that looks as below, and it's as if floodgates have opened and these errors occur all over the place in dozens of different portions of our site. The only way to stop them from happening is to reset the application pool.
We're opening a single session and accompanying transaction for each HTTP request that comes in. It's stored in the HttpContext.Current.Items collection. Each of our repositories extends a base repository class, which references the previously opened session. Here's the module that opens and closes sessions and transactions:
public class NHibernateSessionModule : IHttpModule
{
#region Public Methods
/// <summary>
/// Initializes the http module by hooking up the open session call to the begin
/// request event and the close session call to the end request event.
/// </summary>
/// <param name="context">The context representing the current http request.</param>
public void Init(HttpApplication context)
{
context.BeginRequest += ApplicationBeginRequest;
context.EndRequest += ApplicationEndRequest;
context.Error += OnError;
}
/// <summary>
/// Disposes of the module.
/// </summary>
public void Dispose() { }
#endregion
#region Private Methods
/// <summary>
/// Fired when an error occurs during a request. Cleans up any open sessions and transactions.
/// </summary>
/// <param name="sender">The sender of the event.</param>
/// <param name="e">The event arguments.</param>
private static void OnError(object sender, EventArgs e)
{
var system = HttpContext.Current.Items["SystemSession"] as Lazy<ISession>;
if (system != null && system.IsValueCreated && system.Value != null)
RollbackSession(system.Value);
}
/// <summary>
/// Fired as a request begins. Opens a session and accompanying transaction. Also authenticates the
/// currently logged in user.
/// </summary>
/// <param name="sender">The sender of the event.</param>
/// <param name="e">The event arguments.</param>
private static void ApplicationBeginRequest(object sender, EventArgs e)
{
OpenSession("SystemSession", SessionFactoryProvider.SystemSessionFactory);
}
/// <summary>
/// Opens a new session.
/// </summary>
/// <param name="key">The key used to store the opened session in the current http context.</param>
/// <param name="sessionFactory">The factory used to open the session.</param>
private static void OpenSession(string key, ISessionFactory sessionFactory)
{
if (HttpContext.Current.Items[key] != null)
return;
HttpContext.Current.Items[key] = new Lazy<ISession>(() => {
var session = sessionFactory.OpenSession();
session.BeginTransaction();
return session;
}, true);
}
/// <summary>
/// Fired as a request ends. Closes the previously opened session and commits the session's transaction.
/// </summary>
/// <param name="sender">The sender of the event.</param>
/// <param name="e">The event arguments.</param>
private static void ApplicationEndRequest(object sender, EventArgs e)
{
var system = HttpContext.Current.Items["SystemSession"] as Lazy<ISession>;
if (system != null && system.IsValueCreated && system.Value != null)
CommitSession(system.Value);
}
/// <summary>
/// Commits a session's transaction.
/// </summary>
/// <param name="session">The session.</param>
private static void CommitSession(ISession session)
{
if (session == null)
return;
if (session.Transaction != null && session.Transaction.IsActive)
{
var transaction = session.Transaction;
try
{
transaction.Commit();
}
catch (Exception)
{
transaction.Rollback();
}
}
session.Close();
}
/// <summary>
/// Rolls back a session's transaction.
/// </summary>
/// <param name="session">The session.</param>
private static void RollbackSession(ISession session)
{
if (session == null)
return;
if (session.Transaction != null && session.Transaction.IsActive)
session.Transaction.Rollback();
session.Close();
}
#endregion
}
Here's the static class used to create ISessionFactory instances:
public class SessionFactoryProvider
{
private static Configuration _configuration;
public static Configuration Configuration
{
get
{
return _configuration;
}
}
public static void Initialize()
{
_sessionFactory = Fluently.Configure()
.Database(MsSqlConfiguration.MsSql2008.ConnectionString(c => c.FromConnectionStringWithKey("Database")))
.Mappings(m => m.FluentMappings.AddFromAssemblyOf<CustomerMap>())
.ExposeConfiguration(c => _configuration = c)
.BuildSessionFactory();
}
public static void Initialize(ISessionFactory sessionFactory)
{
_sessionFactory = sessionFactory;
}
private static ISessionFactory _sessionFactory;
public static ISessionFactory SessionFactory
{
get
{
if (_sessionFactory == null)
Initialize();
return _sessionFactory;
}
}
}
Here's the base repository class:
public class BaseRepository<T>
{
#region Properties
/// <summary>
/// Opens and returns a new session.
/// </summary>
private static ISession OpenSession
{
get
{
if (HttpContext.Current != null)
{
var lazySession = (Lazy<ISession>)HttpContext.Current.Items["SystemSession"];
if (lazySession != null)
return lazySession.Value;
}
TypeContainer.Get<ILog>().Warn("Unable to find session in http context");
throw new InvalidOperationException("The current HTTP context contains no system session.");
}
}
private ISession _session;
public ISession CurrentSession
{
get
{
if (_session == null || !_session.IsOpen || _session.Connection.State == ConnectionState.Closed)
_session = OpenSession;
return _session;
}
}
#endregion
}
Here's a sampling of some of the errors we get:
CustomerID20_
at System.Data.ProviderBase.FieldNameLookup.GetOrdinal(String fieldName)
at System.Data.SqlClient.SqlDataReader.GetOrdinal(String name)
at NHibernate.Driver.NHybridDataReader.GetOrdinal(String name)
at NHibernate.Type.NullableType.NullSafeGet(IDataReader rs, String name)
at NHibernate.Type.NullableType.NullSafeGet(IDataReader rs, String[] names, ISessionImplementor session, Object owner)
at NHibernate.Loader.Loader.GetKeyFromResultSet(Int32 i, IEntityPersister persister, Object id, IDataReader rs, ISessionImplementor session)
at NHibernate.Loader.Loader.GetRowFromResultSet(IDataReader resultSet, ISessionImplementor session, QueryParameters queryParameters, LockMode[] lockModeArray, EntityKey optionalObjectKey, IList hydratedObjects, EntityKey[] keys, Boolean returnProxies)
at NHibernate.Loader.Loader.DoQuery(ISessionImplementor session, QueryParameters queryParameters, Boolean returnProxies)
at NHibernate.Loader.Loader.DoQueryAndInitializeNonLazyCollections(ISessionImplementor session, QueryParameters queryParameters, Boolean returnProxies)
at NHibernate.Loader.Loader.DoList(ISessionImplementor session, QueryParameters queryParameters)
The server failed to resume the transaction. Desc:480000000f.
at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection)
at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection)
at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning()
at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)
at System.Data.SqlClient.SqlDataReader.ConsumeMetaData()
at System.Data.SqlClient.SqlDataReader.get_MetaData()
at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)
at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async)
at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, DbAsyncResult result)
at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method)
at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior, String method)
at System.Data.SqlClient.SqlCommand.ExecuteDbDataReader(CommandBehavior behavior)
at System.Data.Common.DbCommand.System.Data.IDbCommand.ExecuteReader()
at NHibernate.AdoNet.AbstractBatcher.ExecuteReader(IDbCommand cmd)
at NHibernate.Loader.Loader.GetResultSet(IDbCommand st, Boolean autoDiscoverTypes, Boolean callable, RowSelection selection, ISessionImplementor session)
at NHibernate.Loader.Loader.DoQuery(ISessionImplementor session, QueryParameters queryParameters, Boolean returnProxies)
at NHibernate.Loader.Loader.DoQueryAndInitializeNonLazyCollections(ISessionImplementor session, QueryParameters queryParameters, Boolean returnProxies)
at NHibernate.Loader.Loader.DoList(ISessionImplementor session, QueryParameters queryParameters)
CustomerID26_
at System.Data.ProviderBase.FieldNameLookup.GetOrdinal(String fieldName)
at System.Data.SqlClient.SqlDataReader.GetOrdinal(String name)
at NHibernate.Driver.NHybridDataReader.GetOrdinal(String name)
at NHibernate.Type.NullableType.NullSafeGet(IDataReader rs, String name)
at NHibernate.Type.NullableType.NullSafeGet(IDataReader rs, String[] names, ISessionImplementor session, Object owner)
at NHibernate.Type.ManyToOneType.Hydrate(IDataReader rs, String[] names, ISessionImplementor session, Object owner)
at NHibernate.Type.ComponentType.Hydrate(IDataReader rs, String[] names, ISessionImplementor session, Object owner)
at NHibernate.Type.ComponentType.NullSafeGet(IDataReader rs, String[] names, ISessionImplementor session, Object owner)
at NHibernate.Loader.Loader.GetKeyFromResultSet(Int32 i, IEntityPersister persister, Object id, IDataReader rs, ISessionImplementor session)
at NHibernate.Loader.Loader.GetRowFromResultSet(IDataReader resultSet, ISessionImplementor session, QueryParameters queryParameters, LockMode[] lockModeArray, EntityKey optionalObjectKey, IList hydratedObjects, EntityKey[] keys, Boolean returnProxies)
at NHibernate.Loader.Loader.DoQuery(ISessionImplementor session, QueryParameters queryParameters, Boolean returnProxies)
at NHibernate.Loader.Loader.DoQueryAndInitializeNonLazyCollections(ISessionImplementor session, QueryParameters queryParameters, Boolean returnProxies)
at NHibernate.Loader.Loader.DoList(ISessionImplementor session, QueryParameters queryParameters)
Transaction not connected, or was disconnected
at NHibernate.Transaction.AdoTransaction.CheckNotZombied()
at NHibernate.Transaction.AdoTransaction.Rollback()
at ACC.Web.Modules.NHibernateSessionModule.CommitSession(ISession session)
at ACC.Web.Modules.NHibernateSessionModule.ApplicationEndRequest(Object sender, EventArgs e)
at System.Web.HttpApplication.SyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)
ID28_0_
at System.Data.ProviderBase.FieldNameLookup.GetOrdinal(String fieldName)
at System.Data.SqlClient.SqlDataReader.GetOrdinal(String name)
at NHibernate.Driver.NHybridDataReader.GetOrdinal(String name)
at NHibernate.Type.NullableType.NullSafeGet(IDataReader rs, String name)
at NHibernate.Type.NullableType.NullSafeGet(IDataReader rs, String[] names, ISessionImplementor session, Object owner)
at NHibernate.Loader.Loader.GetKeyFromResultSet(Int32 i, IEntityPersister persister, Object id, IDataReader rs, ISessionImplementor session)
at NHibernate.Loader.Loader.GetRowFromResultSet(IDataReader resultSet, ISessionImplementor session, QueryParameters queryParameters, LockMode[] lockModeArray, EntityKey optionalObjectKey, IList hydratedObjects, EntityKey[] keys, Boolean returnProxies)
at NHibernate.Loader.Loader.DoQuery(ISessionImplementor session, QueryParameters queryParameters, Boolean returnProxies)
at NHibernate.Loader.Loader.DoQueryAndInitializeNonLazyCollections(ISessionImplementor session, QueryParameters queryParameters, Boolean returnProxies)
at NHibernate.Loader.Loader.DoList(ISessionImplementor session, QueryParameters queryParameters)
Transaction not successfully started
at NHibernate.Transaction.AdoTransaction.CheckBegun()
at NHibernate.Transaction.AdoTransaction.Rollback()
at ACC.Web.Modules.NHibernateSessionModule.CommitSession(ISession session)
at ACC.Web.Modules.NHibernateSessionModule.ApplicationEndRequest(Object sender, EventArgs e)
at System.Web.HttpApplication.SyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)
col_0_0_
at System.Data.ProviderBase.FieldNameLookup.GetOrdinal(String fieldName)
at System.Data.SqlClient.SqlDataReader.GetOrdinal(String name)
at NHibernate.Driver.NHybridDataReader.GetOrdinal(String name)
at NHibernate.Type.NullableType.NullSafeGet(IDataReader rs, String name)
at NHibernate.Type.NullableType.NullSafeGet(IDataReader rs, String[] names, ISessionImplementor session, Object owner)
at NHibernate.Hql.Ast.ANTLR.Loader.QueryLoader.GetResultColumnOrRow(Object[] row, IResultTransformer resultTransformer, IDataReader rs, ISessionImplementor session)
at NHibernate.Loader.Loader.GetRowFromResultSet(IDataReader resultSet, ISessionImplementor session, QueryParameters queryParameters, LockMode[] lockModeArray, EntityKey optionalObjectKey, IList hydratedObjects, EntityKey[] keys, Boolean returnProxies)
at NHibernate.Loader.Loader.DoQuery(ISessionImplementor session, QueryParameters queryParameters, Boolean returnProxies)
at NHibernate.Loader.Loader.DoQueryAndInitializeNonLazyCollections(ISessionImplementor session, QueryParameters queryParameters, Boolean returnProxies)
at NHibernate.Loader.Loader.DoList(ISessionImplementor session, QueryParameters queryParameters)
The server failed to resume the transaction. Desc:8040000001d.
at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection)
at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection)
at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning()
at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)
at System.Data.SqlClient.SqlDataReader.ConsumeMetaData()
at System.Data.SqlClient.SqlDataReader.get_MetaData()
at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)
at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async)
at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, DbAsyncResult result)
at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method)
at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior, String method)
at System.Data.SqlClient.SqlCommand.ExecuteDbDataReader(CommandBehavior behavior)
at System.Data.Common.DbCommand.System.Data.IDbCommand.ExecuteReader()
at NHibernate.AdoNet.AbstractBatcher.ExecuteReader(IDbCommand cmd)
at NHibernate.Loader.Loader.GetResultSet(IDbCommand st, Boolean autoDiscoverTypes, Boolean callable, RowSelection selection, ISessionImplementor session)
at NHibernate.Loader.Loader.DoQuery(ISessionImplementor session, QueryParameters queryParameters, Boolean returnProxies)
Many of the above errors have inner exception messages like:
System.InvalidOperationException: Invalid attempt to call Read when reader is closed.
System.InvalidOperationException: There is already an open DataReader associated with this Command which must be closed first.
System.Data.SqlClient.SqlException: New request is not allowed to start because it should come with valid transaction descriptor.
System.IndexOutOfRangeException: ID28_0_
NHibernate.AssertionFailure: possible non-threadsafe access to the session
System.InvalidOperationException: ExecuteReader requires an open and available Connection. The connection's current state is open.
Lots of these errors appear as though a session is being opened, accessed or closed across multiple threads, but we're not spawning any other threads that access the database anywhere in the application.
Looking through the logs of the NHibernate Profiler tells us that transactions are being opened and committed/rolled back in every case.
We've been trying to fix this problem for some time now and have run out of ideas. Has anyone run into this problem before? Any ideas?
Thanks!
Chris

Answer 1

We're not using NHibernate, so I'm not sure how applicable this is, but we were getting tons of GetOrdinal errors daily. Datareaders and datatables were coming back with unexpected data (i.e. from an entirely different query than we expected). I recently discovered the "Enlist" connection string argument, and after adding "Enlist=False" to all of our connection strings, we've not thrown a single GetOrdinal error.
I believe this change will probably fix alot of the similar issues people have reported here and elsewhere.
Mike