I am trying to improve performance of our nHibernate (3.3.2.4000) application (.NET 4.0). Currently, we are performing CRUD operations one by one, which ends up taking a lot of time, so my plan was to use the ConcurrentQueue and Tasks.
I refactored my code into this:
public void ImportProductsFromXml(string path)
{
List<Product> products = Mapper.GetProducts(path);
var addQueue = new ConcurrentQueue<Product>(productsToAddUpdate);
var updateTasks = new List<Task>();
for (int i = 0; i < 5; i++)
{
var taskId = i + 1;
updateTasks.Add(Task.Factory.StartNew(() => ProcessAddQueue(taskId, products, addQueue)));
}
}
private void ProcessAddQueue(int taskId, List<Product> products, ConcurrentQueue<Product> queue)
{
Product result = null;
while (queue.TryDequeue(out result))
{
try
{
UpdateProducts(products, result);
}
catch (Exception ex)
{
Debug.WriteLine(string.Format("ProcessAddQueue: taskId={0}, SKU={1}, ex={2}", taskId, result.ProductId, ex));
}
}
}
private void UpdateProducts(List<Product> productsFromFile, Product product)
{
...code removed...
CatalogItem parentItem = _catalogRepository.GetByCatalogItemId(category);
...code removed...
_catalogRepository.Save(parentItem);
...code removed...
}
public CatalogItem GetByCatalogItemId(string catalogItemId)
{
using (ISession session = SessionFactory.OpenSession())
{
return session
.CreateCriteria(typeof (CatalogItem))
.Add(Restrictions.Eq("CatalogItemId", catalogItemId))
.List<CatalogItem>().FirstOrDefault();
}
}
The "Save"-method of the catalogRepository calls this method, behind the scenes:
public int Add(T entity)
{
using (ISession session = SessionFactory.OpenSession())
using (ITransaction transaction = session.BeginTransaction())
{
var id = (int) session.Save(entity);
transaction.Commit();
return id;
}
}
So my idea was to create a concurrentqueue containing all the products, and then process them 5 at a time.
However, I am getting an 'Thread was being aborted exception':
at System.WeakReference.get_Target()
at System.Transactions.Transaction.JitSafeGetContextTransaction(ContextData contextData)
at System.Transactions.Transaction.FastGetTransaction(TransactionScope currentScope, ContextData contextData, Transaction& contextTransaction)
at System.Transactions.Transaction.get_Current()
at NHibernate.Transaction.AdoNetWithDistributedTransactionFactory.EnlistInDistributedTransactionIfNeeded(ISessionImplementor session)
at NHibernate.Impl.SessionImpl.get_PersistenceContext()
at NHibernate.Event.Default.DefaultSaveOrUpdateEventListener.EntityIsTransient(SaveOrUpdateEvent event)
at NHibernate.Event.Default.DefaultSaveOrUpdateEventListener.OnSaveOrUpdate(SaveOrUpdateEvent event)
at NHibernate.Impl.SessionImpl.FireSave(SaveOrUpdateEvent event)
at NHibernate.Impl.SessionImpl.Save(Object obj)
What am I doing wrong?
Hibernate sessions are meant to be used as unit of work. You open a session, open a transaction on it, load your entity, modify it, call save, commit/rollback the transaction and then dispose the session.
You should be using ONE session to load your entity and then save it. Currently you are loading an entity with one session and saving it with some other session. Combined with concurrent access this could cause problems.
Try loading and saving the entity with the same hibernate session.
When using hibernate as mentioned it should be fully threadsafe. Please note that a single hibernate session is NOT threadsafe.
Related
I am trying to get a proof of concept running with akka.net. I am sure that I am doing something terribly wrong, but I can't figure out what it is.
I want my actors to form a graph of nodes. Later, this will be a complex graph of business objekts, but for now I want to try a simple linear structure like this:
I want to ask a node for a neighbour that is 9 steps away. I am trying to implement this in a recursive manner. I ask node #9 for a neighbour that is 9 steps away, then I ask node #8 for a neighbour that is 8 steps away and so on. Finally, this should return node #0 as an answer.
Well, my code works, but it takes more than 4 seconds to execute. Why is that?
This is my full code listing:
using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Linq;
using Akka;
using Akka.Actor;
namespace AkkaTest
{
class Program
{
public static Stopwatch stopwatch = new Stopwatch();
static void Main(string[] args)
{
var system = ActorSystem.Create("MySystem");
IActorRef[] current = new IActorRef[0];
Console.WriteLine("Initializing actors...");
for (int i = 0; i < 10; i++)
{
var current1 = current;
var props = Props.Create<Obj>(() => new Obj(current1, Guid.NewGuid()));
var actorRef = system.ActorOf(props, i.ToString());
current = new[] { actorRef };
}
Console.WriteLine("actors initialized.");
FindNeighboursRequest r = new FindNeighboursRequest(9);
stopwatch.Start();
var response = current[0].Ask(r);
FindNeighboursResponse result = (FindNeighboursResponse)response.Result;
stopwatch.Stop();
foreach (var d in result.FoundNeighbours)
{
Console.WriteLine(d);
}
Console.WriteLine("Search took " + stopwatch.ElapsedMilliseconds + "ms.");
Console.ReadLine();
}
}
public class FindNeighboursRequest
{
public FindNeighboursRequest(int distance)
{
this.Distance = distance;
}
public int Distance { get; private set; }
}
public class FindNeighboursResponse
{
private IActorRef[] foundNeighbours;
public FindNeighboursResponse(IEnumerable<IActorRef> descendants)
{
this.foundNeighbours = descendants.ToArray();
}
public IActorRef[] FoundNeighbours
{
get { return this.foundNeighbours; }
}
}
public class Obj : ReceiveActor
{
private Guid objGuid;
readonly List<IActorRef> neighbours = new List<IActorRef>();
public Obj(IEnumerable<IActorRef> otherObjs, Guid objGuid)
{
this.neighbours.AddRange(otherObjs);
this.objGuid = objGuid;
Receive<FindNeighboursRequest>(r => handleFindNeighbourRequest(r));
}
public Obj()
{
}
private async void handleFindNeighbourRequest (FindNeighboursRequest r)
{
if (r.Distance == 0)
{
FindNeighboursResponse response = new FindNeighboursResponse(new IActorRef[] { Self });
Sender.Tell(response, Self);
return;
}
List<FindNeighboursResponse> responses = new List<FindNeighboursResponse>();
foreach (var actorRef in neighbours)
{
FindNeighboursRequest req = new FindNeighboursRequest(r.Distance - 1);
var response2 = actorRef.Ask(req);
responses.Add((FindNeighboursResponse)response2.Result);
}
FindNeighboursResponse response3 = new FindNeighboursResponse(responses.SelectMany(rx => rx.FoundNeighbours));
Sender.Tell(response3, Self);
}
}
}
The reason of such slow behavior is the way you use Ask (an that you use it, but I'll cover this later). In your example, you're asking each neighbor in a loop, and then immediately executing response2.Result which is actively blocking current actor (and thread it resides on). So you're essentially making synchronous flow with blocking.
The easiest thing to fix that, is to collect all tasks returned from Ask and use Task.WhenAll to collect them all, without waiting for each one in a loop. Taking this example:
public class Obj : ReceiveActor
{
private readonly IActorRef[] _neighbours;
private readonly Guid _id;
public Obj(IActorRef[] neighbours, Guid id)
{
_neighbours = neighbours;
_id = id;
Receive<FindNeighboursRequest>(async r =>
{
if (r.Distance == 0) Sender.Tell(new FindNeighboursResponse(new[] {Self}));
else
{
var request = new FindNeighboursRequest(r.Distance - 1);
var replies = _neighbours.Select(neighbour => neighbour.Ask<FindNeighboursResponse>(request));
var ready = await Task.WhenAll(replies);
var responses = ready.SelectMany(x => x.FoundNeighbours);
Sender.Tell(new FindNeighboursResponse(responses.ToArray()));
}
});
}
}
This one is much faster.
NOTE: In general you shouldn't use Ask inside of an actor:
Each ask is allocating a listener inside current actor, so in general using Ask is A LOT heavier than passing messages with Tell.
When sending messages through chain of actors, cost of ask is additionally transporting message twice (one for request and one for reply) through each actor. One of the popular patterns is that, when you are sending request from A⇒B⇒C⇒D and respond from D back to A, you can reply directly D⇒A, without need of passing the message through whole chain back. Usually combination of Forward/Tell works better.
In general don't use async version of Receive if it's not necessary - at the moment, it's slower for an actor when compared to sync version.
I'm using NHibernate to insert some data into Table A. I want to update the status in Table B if the Table A transaction fails. How do I check if it has failed?
Below is my current code:
// Add userId to Receiver
Receiver receiver = new Receiver();
receiver.User = User.GetById(Convert.ToInt32(listItem.Value));
receiver.Notification = Notification.GetById(notification.NotificationId);
receiver.Save();
Where do I call the NHibernate Transaction? If it fails where do I call NHibernate Rollback and update the Table B status?
Take a look at the Official NHibernate documentation on Exception handling: http://nhibernate.info/doc/nh/en/index.html#manipulatingdata-exceptions
using ( ISession session = sessionFactory.OpenSession() )
{
using ( ITransaction transaction = session.BeginTransaction() )
{
try
{
// Do your save/update here for Table A
transaction.Commit();
}
catch( Exception e )
{
// Your save or update failed so this is where you
// could capture that info and update your Table B
transaction.Rollback();
}
}
}
From what I remember, you don't actually have to call tx.Rollback() because when your code leaves the using blocks, it will do that automatically but again, I can't remember exactly. Give it a try and if it doesn't behave as I just described, you can manually rollback in the catch.
using (ISession session = factory.OpenSession())
using (ITransaction tx = session.BeginTransaction())
{
// do some work
tx.Commit();
}
or manually
ISession session = factory.openSession();
try
{
// do some work
session.Flush();
currentTransaction.Commit();
}
catch (Exception e)
{
currentTransaction.Rollback();
throw;
}
finally
{
session.Close();
}
Take a look NHibernate transactions for more details
I have User object with a list of groups that it belongs to:
public UserHeaderMap()
{
Table("USER_HEADER");
Id(x => x.Id, "USER_ID");
HasManyToMany(x => x.Groups)
.Table("USER_GROUP_COMPOSITE")
.ParentKeyColumn("USER_ID")
.ChildKeyColumn("GROUP_ID")
.Cascade.SaveUpdate()
.Inverse();
}
How do I need to modify my mapping or the way I'm retrieving my user object to fill the Groups list upon retrieval? I'm sure there are different options here but I'm not sure which is the best. The collection is currently null whenever I retrieve a User object from the database. I retrieve it using this:
UserHeader userFound = session.Load<UserHeader>(newUser.Id);
Edit:
public class UserHeader
{
public virtual Guid Id { get; set; }
public virtual IList<GroupHeader> Groups { get; set; }
public UserHeader(IList<GroupHeader> groups)
{
Groups = groups;
}
public UserHeader()
{
Groups = new List<GroupHeader>();
}
public override bool Equals(object obj)
{
bool retVal = false;
if (obj is UserHeader)
{
UserHeader otherUser = (UserHeader)obj;
if (Id == otherUser.Id)
retVal = true;
}
return retVal;
}
public override int GetHashCode()
{
return Id.GetHashCode();
}
}
Edit2:
This is the way I was originally querying the data. It's grabbing everything except the many to many relationship.
UserHeader userFound = session.CreateCriteria<UserHeader>()
.Add(Example.Create(newUser))
.UniqueResult<UserHeader>();
Edit3: Unit test
The following unit test fails on foreach (GroupHeader group in userFound.Groups). I can clearly see from the SQL that it is creating the relationship between User and Group in the SQL output. I can post it if necessary.
[TestMethod]
public void CanAddUserToGroup()
{
using (NHibernate.ISession session = SessionOrigin.Current.GetSession())
{
using (NHibernate.ITransaction tran = session.BeginTransaction())
{
session.SaveOrUpdate(newUser);
session.SaveOrUpdate(newGroup);
tran.Commit();
}
newGroup.AddUser(newUser);
using (NHibernate.ITransaction tran = session.BeginTransaction())
{
session.SaveOrUpdate(newGroup);
tran.Commit();
}
GroupHeader groupFound = session.CreateCriteria<GroupHeader>()
.Add(Example.Create(newGroup))
.UniqueResult<GroupHeader>();
UserHeader userFound = session.CreateCriteria<UserHeader>()
.Add(Example.Create(newUser))
.UniqueResult<UserHeader>();
UserHeader userFound2 = session.Load<UserHeader>(newUser.Id);
Assert.IsNotNull(groupFound, "Failed to find group after insertion");
Assert.IsNotNull(userFound, "Failed to find user after insertion");
UserHeader userInGroup = null;
GroupHeader groupInUser = null;
foreach (UserHeader user in groupFound.Users)
{
if (user.Equals(newUser))
userInGroup = user;
}
foreach (GroupHeader group in userFound.Groups)
{
if (group.Equals(newGroup))
groupInUser = group;
}
Assert.IsNotNull(userInGroup, "Failed to add a new user to group");
Assert.IsNotNull(groupInUser, "Failed to add a new group to a user");
using (NHibernate.ITransaction tran = session.BeginTransaction())
{
session.Delete(newUser);
session.Delete(newGroup);
tran.Commit();
}
}
}
Your unit test code is not correct.
Since both the group and the user are already loaded in the session, the criteria queries return the same instances.
If the collections were null in memory, they'll still be null after the query. Also, you're adding to just one side of a bidirectional association.
Last but not least, you are testing too many unrelated things in the same test method.
I need to retrieve multiple objects from an external system. The external system supports multiple simultaneous requests (i.e. threads), but it is possible to flood the external system - therefore I want to be able to retrieve multiple objects asynchronously, but I want to be able to throttle the number of simultaneous async requests. i.e. I need to retrieve 100 items, but don't want to be retrieving more than 25 of them at once. When each request of the 25 completes, I want to trigger another retrieval, and once they are all complete I want to return all of the results in the order they were requested (i.e. there is no point returning the results until the entire call is returned). Are there any recommended patterns for this sort of thing?
Would something like this be appropriate (pseudocode, obviously)?
private List<externalSystemObjects> returnedObjects = new List<externalSystemObjects>;
public List<externalSystemObjects> GetObjects(List<string> ids)
{
int callCount = 0;
int maxCallCount = 25;
WaitHandle[] handles;
foreach(id in itemIds to get)
{
if(callCount < maxCallCount)
{
WaitHandle handle = executeCall(id, callback);
addWaitHandleToWaitArray(handle)
}
else
{
int returnedCallId = WaitHandle.WaitAny(handles);
removeReturnedCallFromWaitHandles(handles);
}
}
WaitHandle.WaitAll(handles);
return returnedObjects;
}
public void callback(object result)
{
returnedObjects.Add(result);
}
Consider the list of items to process as a queue from which 25 processing threads dequeue tasks, process a task, add the result then repeat until the queue is empty:
class Program
{
class State
{
public EventWaitHandle Done;
public int runningThreads;
public List<string> itemsToProcess;
public List<string> itemsResponses;
}
static void Main(string[] args)
{
State state = new State();
state.itemsResponses = new List<string>(1000);
state.itemsToProcess = new List<string>(1000);
for (int i = 0; i < 1000; ++i)
{
state.itemsToProcess.Add(String.Format("Request {0}", i));
}
state.runningThreads = 25;
state.Done = new AutoResetEvent(false);
for (int i = 0; i < 25; ++i)
{
Thread t =new Thread(new ParameterizedThreadStart(Processing));
t.Start(state);
}
state.Done.WaitOne();
foreach (string s in state.itemsResponses)
{
Console.WriteLine("{0}", s);
}
}
private static void Processing(object param)
{
Debug.Assert(param is State);
State state = param as State;
try
{
do
{
string item = null;
lock (state.itemsToProcess)
{
if (state.itemsToProcess.Count > 0)
{
item = state.itemsToProcess[0];
state.itemsToProcess.RemoveAt(0);
}
}
if (null == item)
{
break;
}
// Simulate some processing
Thread.Sleep(10);
string response = String.Format("Response for {0} on thread: {1}", item, Thread.CurrentThread.ManagedThreadId);
lock (state.itemsResponses)
{
state.itemsResponses.Add(response);
}
} while (true);
}
catch (Exception)
{
// ...
}
finally
{
int threadsLeft = Interlocked.Decrement(ref state.runningThreads);
if (0 == threadsLeft)
{
state.Done.Set();
}
}
}
}
You can do the same using asynchronous callbacks, there is no need to use threads.
Having some queue-like structure to hold the pending requests is a pretty common pattern. In Web apps where there may be several layers of processing you see a "funnel" style approach with the early parts of the processing change having larger queues. There may also be some kind of prioritisation applied to queues, higher priority requests being shuffled to the top of the queue.
One important thing to consider in your solution is that if request arrival rate is higher than your processing rate (this might be due to a Denial of Service attack, or just that some part of the processing is unusually slow today) then your queues will increase without bound. You need to have some policy such as to refuse new requests immediately when the queue depth exceeds some value.
I've created an integration test to verify that a repository handles Concurrency correcly. If I run the test without a TransactionScope, everything works as expect, but if I wrap the test in a TransactionScope, I get an error suggesting that there is a sudden need for distributed transactions (which lead me to believe that there is a second transaction being created). Here is the test:
[Test]
public void Commit_ItemToCommitContainsStaleData_ThrowsStaleObjectStateException()
{
using (new TransactionScope())
{
// arrange
RootUnitOfWorkFactory factory = CreateUnitOfWorkFactory();
const int Id = 1;
WorkItemRepository firstRepository = new WorkItemRepository(factory);
WorkItem itemToChange = WorkItem.Create(Id);
firstRepository.Commit(itemToChange);
WorkItemRepository secondRepository = new WorkItemRepository(factory);
WorkItem copyOfItemToChange = secondRepository.Get(Id);
// act
copyOfItemToChange.ChangeDescription("A");
secondRepository.Commit(copyOfItemToChange);
itemToChange.ChangeDescription("B");
// assert
Assert.Throws<StaleObjectStateException>(() => firstRepository.Commit(itemToChange));
}
}
This is the bottom of the error stack:
failed: NHibernate.Exceptions.GenericADOException : could not load an entity: [TfsTimeMachine.Domain.WorkItem#1][SQL: SELECT workitem0_.Id as Id1_0_, workitem0_.LastChanged as LastChan2_1_0_, workitem0_.Description as Descript3_1_0_ FROM [WorkItem] workitem0_ WHERE workitem0_.Id=?]
----> System.Data.SqlClient.SqlException : MSDTC on server 'ADM4200\SQLEXPRESS' is unavailable.
at NHibernate.Loader.Loader.LoadEntity(ISessionImplementor session, Object id, IType identifierType, Object optionalObject, String optionalEntityName, Object optionalIdentifier, IEntityPersister persister).
I'm running NUnit 2.1, so can someone tell me if Nhibernate creates implicit transactions if there is no session.BeginTransaction() before querying data, regardless of the session running within a TransactionScope?
I got this to work. The problem was (as stated in my comment) that two concurrent sessions were started within the same transactionscope and both started a new dbconnection which enlisted the same transaction, thus forcing DTC to kick in. The solution to this was to create a custom connection provider which ensured that the same connection was returned while inside a transactionscope. I then put this into play in my test and presto, I could test stale object state and rollback the data when the tests completes. Heres my implementation:
/// <summary>
/// A connection provider which returns the same db connetion while
/// there exists a TransactionScope.
/// </summary>
public sealed class AmbientTransactionAwareDriverConnectionProvider : IConnectionProvider
{
private readonly bool disposeDecoratedProviderWhenDisposingThis;
private IConnectionProvider decoratedProvider;
private IDbConnection maintainedConnectionThroughAmbientSession;
public AmbientTransactionAwareDriverConnectionProvider()
: this(new DriverConnectionProvider(), true)
{}
public AmbientTransactionAwareDriverConnectionProvider(IConnectionProvider decoratedProvider,
bool disposeDecoratedProviderWhenDisposingThis)
{
Guard.AssertNotNull(decoratedProvider, "decoratedProvider");
this.decoratedProvider = decoratedProvider;
this.disposeDecoratedProviderWhenDisposingThis = disposeDecoratedProviderWhenDisposingThis;
}
~AmbientTransactionAwareDriverConnectionProvider()
{
Dispose(false);
}
public void Dispose()
{
Dispose(true);
GC.SuppressFinalize(this);
}
public void Configure(IDictionary<string, string> settings)
{
this.decoratedProvider.Configure(settings);
}
public void CloseConnection(IDbConnection conn)
{
if (Transaction.Current == null)
this.decoratedProvider.CloseConnection(conn);
}
public IDbConnection GetConnection()
{
if (Transaction.Current == null)
{
if (this.maintainedConnectionThroughAmbientSession != null)
this.maintainedConnectionThroughAmbientSession.Dispose();
return this.decoratedProvider.GetConnection();
}
if (this.maintainedConnectionThroughAmbientSession == null)
this.maintainedConnectionThroughAmbientSession = this.decoratedProvider.GetConnection();
return this.maintainedConnectionThroughAmbientSession;
}
private void Dispose(bool disposing)
{
if (this.maintainedConnectionThroughAmbientSession != null)
CloseConnection(this.maintainedConnectionThroughAmbientSession);
if (this.disposeDecoratedProviderWhenDisposingThis && this.decoratedProvider != null)
this.decoratedProvider.Dispose();
if (disposing)
{
this.decoratedProvider = null;
this.maintainedConnectionThroughAmbientSession = null;
}
}
public IDriver Driver
{
get { return this.decoratedProvider.Driver; }
}
}
I'm not sure if Hibernate is using transactions internally, but I also don't think that is your problem here.
It appears that the problem is that you are using two different data sources in the same transaction. In order to coordinate the transaction between both data sources for a two-phase commit, you would need to have DTC enabled. The fact that both data sources are actually the same database is immaterial.