I wrote a test case to test the integrity of the Object Store managed by Objectify:
#Test
public void testCreateDuplicateWithDiffID() throws NullValueException, NotFoundException {
gift.setGiftedOn(Calendar.getInstance().getTime());
gEndPt.create(gift);
int countBefore2ndCreate = OfyController.count(Gift.class);
// reinitialize the instanceID of the gift...now it's "different"
gift.initID();
// got to make sure we have a new date
gift.setGiftedOn(Calendar.getInstance().getTime());
// attempt to "re-create" the same instance
gEndPt.create(gift);
int countAfterCreate = OfyController.count(Gift.class);
// check the count...make sure it changes after the two inserts
assertThat("New Gift was NOT created", countAfterCreate, is(2));
QueryResultIterator<Gift> iterator = OfyController.ofy().load().type(Gift.class).iterator();
while (iterator.hasNext()){
log.info(iterator.next().getInstID());
}
}
Everything is fine up to the while loop. For some strange reason, the iterator is displaying the same record (don't know it's just not iterating through or the same record was created twice)
It's hard to know exactly what's going on here but one thing you should never do is create an entity, save it, then try to change its id. Entity instances are saved in the session cache so you're probably confusing the hell out of the cache.
There is no reason to recycle object instances like this. You should never reset the id of an entity instance.
Related
I have a bunch of existing sagas in various states of a long running process.
Recently we decided to make one of the properties on our IContainSagaData implementation unique by using the Saga.UniqueAttribute (about which more here http://docs.particular.net/nservicebus/nservicebus-sagas-and-concurrency).
After deploying the change, we realized that all our old saga instances were not being found, and after further digging (thanks Charlie!) discovered that by adding the unique attribute, we were required to data fix all our existing sagas in Raven.
Now, this is pretty poor, kind of like adding a index to a database column and then finding that all the table data no longer select-able, but being what it is, we decided to create a tool for doing this.
So after creating and running this tool we've now patched up the old sagas so that they now resemble the new sagas (sagas created since we went live with the change).
However, despite all the data now looking right we're still not able to find old instances of the saga!
The tool we wrote does two things. For each existing saga, the tool:
Adds a new RavenJToken called "NServiceBus-UniqueValue" to the saga metadata, setting the value to the same value as our unique property for that saga, and
Creates a new document of type NServiceBus.Persistence.Raven.SagaPersister.SagaUniqueIdentity, setting the SagaId, SagaDocId, and UniqueValue fields accordingly.
My questions are:
Is it sufficient to simply make the data look correct or is there something else we need to do?
Another option we have is to revert the change which added the unique attribute. However in this scenario, would those new sagas which have been created since the change went in be OK with this?
Code for adding metadata token:
var policyKey = RavenJToken.FromObject(saga.PolicyKey); // This is the unique field
sagaDataMetadata.Add("NServiceBus-UniqueValue", policyKey);
Code for adding new doc:
var policyKeySagaUniqueId = new SagaUniqueIdentity
{
Id = "Matlock.Renewals.RenewalSaga.RenewalSagaData/PolicyKey/" + Guid.NewGuid().ToString(),
SagaId = saga.Id,
UniqueValue = saga.PolicyKey,
SagaDocId = "RenewalSaga/" + saga.Id.ToString()
};
session.Store(policyKeySagaUniqueId);
Any help much appreciated.
EDIT
Thanks to David's help on this we have fixed our problem - the key difference was we used the SagaUniqueIdentity.FormatId() to generate our document IDs rather than a new guid - this was trivial tio do since we were already referencing the NServiceBus and NServiceBus.Core assemblies.
The short answer is that it is not enough to make the data resemble the new identity documents. Where you are using Guid.NewGuid().ToString(), that data is important! That's why your solution isn't working right now. I spoke about the concept of identity documents (specifically about the NServiceBus use case) during the last quarter of my talk at RavenConf 2014 - here are the slides and video.
So here is the long answer:
In RavenDB, the only ACID guarantees are on the Load/Store by Id operations. So if two threads are acting on the same Saga concurrently, and one stores the Saga data, the second thread can only expect to get back the correct saga data if it is also loading a document by its Id.
To guarantee this, the Raven Saga Persister uses an identity document like the one you showed. It contains the SagaId, the UniqueValue (mostly for human comprehension and debugging, the database doesn't technically need it), and the SagaDocId (which is a little duplication as its only the {SagaTypeName}/{SagaId} where we already have the SagaId.
With the SagaDocId, we can use the Include feature of RavenDB to do a query like this (which is from memory, probably wrong, and should only serve to illustrate the concept as pseudocode)...
var identityDocId = // some value based on incoming message
var idDoc = RavenSession
// Look at the identity doc's SagaDocId and pull back that document too!
.Include<SagaIdentity>(identityDoc => identityDoc.SagaDocId)
.Load(identityDocId);
var sagaData = RavenSession
.Load(idDoc.SagaDocId); // Already in-memory, no 2nd round-trip to database!
So then the identityDocId is very important because it describes the uniqueness of the value coming from the message, not just any old Guid will do. So what we really need to know is how to calculate that.
For that, the NServiceBus saga persister code is instructive:
void StoreUniqueProperty(IContainSagaData saga)
{
var uniqueProperty = UniqueAttribute.GetUniqueProperty(saga);
if (!uniqueProperty.HasValue) return;
var id = SagaUniqueIdentity.FormatId(saga.GetType(), uniqueProperty.Value);
var sagaDocId = sessionFactory.Store.Conventions.FindFullDocumentKeyFromNonStringIdentifier(saga.Id, saga.GetType(), false);
Session.Store(new SagaUniqueIdentity
{
Id = id,
SagaId = saga.Id,
UniqueValue = uniqueProperty.Value.Value,
SagaDocId = sagaDocId
});
SetUniqueValueMetadata(saga, uniqueProperty.Value);
}
The important part is the SagaUniqueIdentity.FormatId method from the same file.
public static string FormatId(Type sagaType, KeyValuePair<string, object> uniqueProperty)
{
if (uniqueProperty.Value == null)
{
throw new ArgumentNullException("uniqueProperty", string.Format("Property {0} is marked with the [Unique] attribute on {1} but contains a null value. Please make sure that all unique properties are set on your SagaData and/or that you have marked the correct properties as unique.", uniqueProperty.Key, sagaType.Name));
}
var value = Utils.DeterministicGuid.Create(uniqueProperty.Value.ToString());
var id = string.Format("{0}/{1}/{2}", sagaType.FullName.Replace('+', '-'), uniqueProperty.Key, value);
// raven has a size limit of 255 bytes == 127 unicode chars
if (id.Length > 127)
{
// generate a guid from the hash:
var key = Utils.DeterministicGuid.Create(sagaType.FullName, uniqueProperty.Key);
id = string.Format("MoreThan127/{0}/{1}", key, value);
}
return id;
}
This relies on Utils.DeterministicGuid.Create(params object[] data) which creates a Guid out of an MD5 hash. (MD5 sucks for actual security but we are only looking for likely uniqueness.)
static class DeterministicGuid
{
public static Guid Create(params object[] data)
{
// use MD5 hash to get a 16-byte hash of the string
using (var provider = new MD5CryptoServiceProvider())
{
var inputBytes = Encoding.Default.GetBytes(String.Concat(data));
var hashBytes = provider.ComputeHash(inputBytes);
// generate a guid from the hash:
return new Guid(hashBytes);
}
}
}
That's what you need to replicate to get your utility to work properly.
What's really interesting is that this code made it all the way to production - I'm surprised you didn't run into trouble before this, with messages creating new saga instances when they really shouldn't because they couldn't find the existing Saga data.
I almost think it might be a good idea if NServiceBus would raise a warning any time you tried to find Saga Data by anything other than a [Unique] marked property, because it's an easy thing to forget to do. I filed this issue on GitHub and submitted this pull request to do just that.
Ok, each and every time I get into this situation, I struggle back and forth until I find a way to solve it (and that is usually not the way I would have liked to solve it).
What I'm talking about is disconnected entities in EF that should update existing entities in the database.
I'll give an example of my problem here (this example is the last time I got into this problem that caused me to write this question).
I have a WCF service that uses Entity Framework as well. The other program that have added a service reference to my service have gotten proxy versions of the Entities as normal.
The case is that the consumer of the service now construct a object of this proxy class, and call the method UpdateEntity on the WCF service. This entity has a foreign key to another type of entities, and the primary key of the entity I want to link this new entity to is also sent as a parameter to this method. In this case, I want the entity with the same primary key in the database to be updated. It seems simple enough right?
My method looks something like this now:
public bool ChangeEntity(MyEntity entity, int otherTableForignKey)
{
//first I verify that the entity to update exist in the system
var entitytochange = entityContext.MyEntities.FirstOrDefault(e => e.Name == entity.Name);
if (systemtochange == null) return false;
try
{
entity.ForignEntity = entityContext.ForeignEntities.FirstOrDefault(f => f.Key == otherTableForignKey);
//code for updating the entity should go here, but I'm nor sure what
entityContext.SaveChanges();
return true;
}
catch (Exception exc)
{
return false;
}
}
I tried many different combinations of ApplyCurrentValues, Attach, setting ObjectState to Modified and so on, but I get either the error message that I can't add a new entity with the same key as an existing entity, that the object state of the new object can't be Added and so on.
So my question is: What is the best way to do this without writing code that looks like a big hack.
The only way I got this working now was to just set the properties of entitytochange manually with the properties of entity, but it is a bad solution since any added properties to MyEntity will break the code if I don't remember to add code in this method as well, and it seems there really should be another way that is better.
EDIT
When I put entityContext.MyEntities.ApplyCurrentValues(entity); where my comment is put above, I get the following exception on this line:
The existing object in the ObjectContext is in the Added state. Changes can only be applied when the existing object is in an unchanged or modified state.
However, if I remove this line above entity.ForignEntity = entityContext.ForeignEntities.FirstOrDefault(f => f.Key == otherTableForignKey); then the ApplyCurrentValues works without any problems.
Why would me setting the ForeignEntity of the object set it to Added state? So it seems that setting a Property on the Detached entity, attaches it to the context with a state of added?
The following scenario is, I would say quite common and although I know one way of resolving it but it lack elegance.
The example I'm giving is based upon https://github.com/sharparchitecture/Sharp-Architecture-Cookbook.
The application I'm coding is an ASP.NET MVC application and has to support multiple users working on the same object.
The following scenario is an edge case but nevertheless a valid one.
Say you have two users working on the same object and whether the dB row can be updated depends upon the value of a particular field. To make it more concrete, let's say you have a Product and to keep things simple, this Product has "Name" and "QuantityInStock" fields.
Say that initially, there are 10 items of the Product and User1 and User2 want to buy this product. When both users are presented the initial form they are told that there are 10 of these items in stock. Now User1 buys all 10 items while User2 goes to have a coffee. So User1's transaction goes through no problem.
Then User2 comes back after his coffee in the belief that there are still 10 items in stock. So he tries to buy 1 but he must be prevented from doing so since there are no items in stock.
So this problem can be solved by using ASP.NET DataAnnotations validation and this will catch the majority of cases. However, in our edge case, say that User1 and User2 perform the same operation but within a fraction of a second such that when User2 submits the form, it passes the ASP.NET Validation but by the time it gets to the persistence layer, the QuantityInStock is 0.
The solution to this is to perform the validation at the latest moment as possible i.e. just before calling the Update method.
Now for some code.
public ProductModel CreateOrUpdate(ProductModel productModel)
{
var currentProductModel = Get(productModel.Id);
var currentQuantityInStock = currentProductModel.QuantityInStock;
if(currentProductModel.QuantityInStock !=0 && productModel.QuantityInStock >= currentQuantityInStock )
{
currentProductModel.QuantityInStock= productModel.QuantityInStock;
currentProductModel.Name = productModel.Name;
this.productModelRepository.SaveOrUpdate( currentProductModel );
return productModel;
}
else
{
//Raise an exception
}
}
Now, the fact that I'm calling:
var currentProductModel = Get(productModel.Id);
means that if I were to just do this:
this.productModelRepository.SaveOrUpdate( productModel );
would cause an exception:
a different object with the same identifier value was already associated with the session: 1
Hence, I have to copy all of the values from productModel to currentProductModel. It's fine when using something like Automapper but still kind of feels wrong to me in the sense that I feel I should just be able to save productModel as is without having to transfer the data from one object to another.
Moreover, having to do the same validation twice, once using DataAnnotation and another time just before updating violates the DRY principle.
The point is that I feel like I'm missing a trick but don't quite know where to start and what to investigate.
This to me is a simple problem but coming up with a nice elegant solution is something else. So the question is how have you dealt with this simple case in the past? Am I overthinking this?
have you tried optimistic Locking with Version?
// Fluent mapping
public EntitiyMap()
{
OptimisticLock.All(); // all properties musn't be changed in db when saving
// or
OptimisticLock.Dirty(); // only dirty properties musn't be changed in db when saving
}
//
public ProductModel CreateOrUpdate(ProductModel productModel)
{
try
{
// productModel is already validated and updated
this.productModelRepository.SaveOrUpdate( productModel );
return productModel;
}
catch (StaleObjectException)
{
// somebody changed the object in database after we have read it
// Raise an exception or whatever
}
}
Update: i handled such things in another way
public void BuySomething(ProductModel productModel, int amount)
{
int tries = 5;
bool success = false;
while(!success && tries > 0)
{
if (productModel.QuantityInStock <= amount)
{
//Raise an exception
}
productModel.QuantityInStock - amount;
try
{
this.productModelRepository.SaveOrUpdate( productModel );
}
catch (StaleObjectException)
{
// somebody changed the object in database after we have read it
this.productModelRepository.Refresh(productModel);
tries--;
}
}
if (tries <= 0)
{
// Raise an exception or whatever
}
}
zero extra roundtrips if nobody changed it in between, and guaranteed serialisation of the transactions
I'm developing a WCF Data Service with self tracking entities and I want to prevent clients from inserting duplicated content. Whenever they POST data without providing a value for the data key, I have to execute some logic to determine whether that data is already present inside my database or not. I've written a Change interceptor like this:
[ChangeInterceptor("MyEntity")]
public void OnChangeEntity(MyEntity item, UpdateOperations operations){
if (operations == UpdateOperations.Add)
{
// Here I search the database to see if a matching record exists.
// If a record is found, I'd like to use its ID and basically change an insertion
// into an update.
item.EntityID = existingEntityID;
item.MarkAsModified();
}
}
However, this is not working. The existingEntityID is ignored and, as a result, the record is always inserted, never updated. Is it even possible to do? Thanks in advance.
Hooray! I managed to do it.
item.EntityID = existingEntityID;
this.CurrentDataSource.ObjectStateManager.ChangeObjectState(item, EntityState.Modified);
I had to change the object state elsewhere, ie. by calling .ChangeObjectState of the ObjectStateManager, which is a property of the underlying EntityContext. I was mislead by the .MarkAsModified() method which, at this point, I'm not sure what it does.
With my only ORM knowledge being L2S/EF, I was surprised when the following code inserted a row into the database before I called repo.Save:
var repo = new UserRepository();
var user = new User { Name = "test" }
repo.Add(user);
//repo.Save();
Repo looks like this:
public void Add(T entity)
{
session.Save(entity);
}
public void Save()
{
session.Flush();
}
After some digging, it seems NHibernate needs to make the insert happen right away in order to get the ID of the new entity (since it's using an auto increment ID). But L2S/EF doesn't work like this; I can add many entities and save them all at the end.
Question is: is there a way to achieve the same thing with NHibernate, while still using auto increment IDs, and out of interest does anyone know why it works like this?
Fabio Maulo already blogged about the usage of identity generator a few times. The answer is: use hilo, guid.comb or something like this.
NHibernate needs the identity because every entity in the session (they are called "persistent entities") needs to be identified. The identity is also normally used to determine if the record already exists in the database (unsaved value).
session.Save actually only makes a transient entity persistent. When the database is generating the id, it needs to be stored to get the id. If NH can create the id itself (eg using hilo), it could be stored next time when the session gets flushed.