I am trying to get a grip on the ServiceStack Redis example and Redis itself and now have some questions.
Question 1:
I see some static indexes defined, eg:
static class TagIndex
{
public static string Questions(string tag) { return "urn:tags>q:" + tag.ToLower(); }
public static string All { get { return "urn:tags"; } }
}
What does that '>' (greater than) sign do? Is this some kind of convention?
Question 2:
public User GetOrCreateUser(User user)
{
var userIdAliasKey = "id:User:DisplayName:" + user.DisplayName.ToLower();
using (var redis = RedisManager.GetClient())
{
var redisUsers = redis.As<User>();
var userKey = redis.GetValue(userIdAliasKey);
if (userKey != null) return redisUsers.GetValue(userKey);
if (user.Id == default(long)) user.Id = redisUsers.GetNextSequence();
redisUsers.Store(user);
redis.SetEntry(userIdAliasKey, user.CreateUrn());
return redisUsers.GetById(user.Id);
}
}
As far as I can understand, first a user is stored with a unique id. Is this necessary when using the client (I know this is not for Redis necessary)? I have for my model a meaningful string id (like an email address) which I like to use. I also see a SetEntry is done. What does SetEntry do exactly? I think it is an extra key just to set a relation between the id and a searchable key. I guess this is not necessary when storing the object itself with a meaningful key, so user.id = "urn:someusername". And how is SetEntry stored as a Redis Set or just an extra key?
Question 3:
This is more Redis related but I am trying to figure out how everything is stored in Redis in order to get a grip on the example so I did:
Started redis-cli.exe in a console
Typed 'keys *' this shows all keys
Typed 'get id:User:DisplayName:joseph' this showed 'urn:user:1'
Typed 'get urn:user:1' this shows the user
Now I also see keys like 'urn:user>q:1' or 'urn:tags' if I do a 'get urn:tags' I get the error 'ERR Operation against a key holding the wrong kind of value'. And tried other Redis commands like smembers but I cannot find the right query commands.
Question 1: return "urn:tags>q:" + tag.ToLower(); gives you the key (a string) for a given tag; the ">" has no meaning for Redis, it's a convention of the developer of the example, and could have been any other character.
Question 3: use the TYPE command to determine the type of the key, then you'll find the right command in redis documentation to get the values.
Related
When the key is a composite of id1, id2 in a GemFire Region and the Region is partitioned with id1, what is the best way of getting all the rows whose key matched id1.
Couple of options that we are thinking of:
Create another index on id1. If we do that, we are wondering if it goes against all Partition Regions?
Write data aware Function and Filter by (id1, null) to target specific Partition Region. Use index in local Region by using QueryService?
Can you please let me know if there is any other way to achieve the query by partial key.
Well, it could be implemented (optimally) by using a combination of #1 and #2 in your "options" above (depending on whether your application domain object also stored/referenced the key, which would be the case if you were using SD[G] Repositories.
This might be best explained with the docs and an example, particularly using the PartitionResolver interface Javadoc.
Say your "composite" Key was implemented as follows:
class CompositeKey implements PartitionResolver {
private final Object idOne;
private final Object idTwo;
CompositeKey(Object idOne, Object idTwo) {
// argument validation as necessary
this.idOne = idOne;
this.idTwo = idTwo;
}
public String getName() {
return "MyCompositeKeyPartitionResolver";
}
public Object getRoutingObject() {
return idOne;
}
}
Then, you could invoke a Function that queries the results you desire by using...
Execution execution = FunctionService.onRegion("PartitionRegionName");
Optionally, you could use the returned Execution to filter on just the (complex) Keys you wanted to query (further qualify) when invoking the Function...
ComplexKey filter = { .. };
execution.withFilter(Arrays.stream(filter).collect(Collectors.toSet()));
Of course, this is problematic if you do not know your keys in advance.
Then you might prefer to use the ComplexKey to identify your application domain object, which is necessary when using SD[G]'s Repository abstraction/extension:
#Region("MyPartitionRegion")
class ApplicationDomainObject {
#Id
CompositeKey identifier;
...
}
And then, you can code your Function to operate on the "local data set" of the Partition Region. That is, when a data node in the cluster hosts the same Partition Region (PR), then it will only operate on the data set in the "bucket" for that PR, which is accomplished by doing the following:
class QueryPartitionRegionFunction implements Function {
public void execute(FunctionContext<Object> functionContext) {
RegionFunctionContext regionFunctionContext =
(RegionFunctionContext) functionContext;
Region<ComplexKey, ApplicationDomainObject> localDataSet =
PartitionRegionHelper.getLocalDataForContext(regionFunctionContext);
SelectResults<?> resultSet =
localDataSet.query(String.format("identifier.idTwo = %s",
regionFunctionContext.getArguments);
// process result set and use ResultSender to send results
}
}
Of course, all of this is much easier to do using SDG's Function annotation support (i.e. implementing and invoking your Function anyway).
Note that, when you invoke the Function, onRegion using the GemFire's FunctionService, or more conveniently with SDG's annotation support for Function Execution, like so:
#OnRegion("MyPartitionRegion")
interface MyPartitionRegionFunctions {
#FunctionId("QueryPartitionRegion")
<return-type> queryPartitionRegion(..);
}
Then..
Object resultSet = myPartitionRegionFunctions.queryPartitionRegion(..);
Then, the FunctionContext will be a RegionFunctionContext (because you executed the Function on the PR, which executes on all nodes in the cluster hosting the PR).
Additionally, you use the PartitionRegionHelper.getLocalDataForContext(:RegionFunctionContext) to get the local data set of the PR (i.e. the bucket, or just the shard of data in the entire PR (across all nodes) hosted by that node, which would be based your "custom" PartitionResolver).
You can then query to further qualify, or filter the data of interests. You can see that I queried (or further qualified) by idTwo, which was not part of the PartitionResolver implementation. Additionally, this would only be required in the (OQL) query predicate if you did not specify Keys in your Filter with the Execution (since, I think, that would take the entire "Key" (idOne & idTwo) into account, based on our properly implemented Object.equals() method of your ComplexKey class).
But, if you did not know the keys in advance and/or (especially if) you are using SD[G]'s Repositories, then the ComplexKey would be part of your application domain abject, which you could then Index, and query on (as shown above: identifier.idTwo = ?).
Hope this helps!
NOTE: I have not test any of this, but hopefully it will point you in the right direction and/or give you further ideas.
[Table("tblClients")]
public class ClientsTbl
{
[DatabaseGeneratedAttribute(DatabaseGeneratedOption.Identity)]
public int ClientID { get; set; }
[Key]
public int userid { get; set; }
}
If i have old data (from unknown source) to import which has userid that related with other tables for example invoice etc without using Foreign Key, how can i design my DB model? If i would design like that insert must be like that :
Random a = new Random();
ClientsTbl c = new ClientsTbl()
{
userid = a.Next(-1000, -1),
}
dataContext.Clients.Add(c);
dataContext.SaveChanges();
var client = dataContext.Clients.FirstOrDefault(x => x.userid == c.userid);
if(client!=null)
{
client.userid = c.ClientID;
}
dataContext.SaveChanges();
after inserting a record I need to update "userid" : `` And it gives an error
The property 'userid' is part of the object's key information and cannot be modified.
If i would change all operations to Stored Procedure's (insert and update) everything is ok.
Why Entity Framework doesn't allow to update Key ? (Sql server does)
This is due to the fact that you're referencing the same object. This has been asked before and answered here:
The property 'Id' is part of the object's key information and cannot be modified
I hope this helps you.
OO paradigm has this thing about "object identity". RM does not have this (not in the sense that every identifier of a thing MUST MANDATORILY remain unchanged during the whole lifetime of the thing it identifies).
Tools that attempt to bridge the gap between the two are therefore inevitably faced with a kind of mismatch, and must do one of two things to address it : (a) break the OO paradigm of cast-in-concrete-immutable identity, or (b) enforce the OO paradigm and rule out perfectly legitimate RM operations.
Most if not all of them opt for (b) because that is the easiest way out, at least for the developers.
I have a bunch of existing sagas in various states of a long running process.
Recently we decided to make one of the properties on our IContainSagaData implementation unique by using the Saga.UniqueAttribute (about which more here http://docs.particular.net/nservicebus/nservicebus-sagas-and-concurrency).
After deploying the change, we realized that all our old saga instances were not being found, and after further digging (thanks Charlie!) discovered that by adding the unique attribute, we were required to data fix all our existing sagas in Raven.
Now, this is pretty poor, kind of like adding a index to a database column and then finding that all the table data no longer select-able, but being what it is, we decided to create a tool for doing this.
So after creating and running this tool we've now patched up the old sagas so that they now resemble the new sagas (sagas created since we went live with the change).
However, despite all the data now looking right we're still not able to find old instances of the saga!
The tool we wrote does two things. For each existing saga, the tool:
Adds a new RavenJToken called "NServiceBus-UniqueValue" to the saga metadata, setting the value to the same value as our unique property for that saga, and
Creates a new document of type NServiceBus.Persistence.Raven.SagaPersister.SagaUniqueIdentity, setting the SagaId, SagaDocId, and UniqueValue fields accordingly.
My questions are:
Is it sufficient to simply make the data look correct or is there something else we need to do?
Another option we have is to revert the change which added the unique attribute. However in this scenario, would those new sagas which have been created since the change went in be OK with this?
Code for adding metadata token:
var policyKey = RavenJToken.FromObject(saga.PolicyKey); // This is the unique field
sagaDataMetadata.Add("NServiceBus-UniqueValue", policyKey);
Code for adding new doc:
var policyKeySagaUniqueId = new SagaUniqueIdentity
{
Id = "Matlock.Renewals.RenewalSaga.RenewalSagaData/PolicyKey/" + Guid.NewGuid().ToString(),
SagaId = saga.Id,
UniqueValue = saga.PolicyKey,
SagaDocId = "RenewalSaga/" + saga.Id.ToString()
};
session.Store(policyKeySagaUniqueId);
Any help much appreciated.
EDIT
Thanks to David's help on this we have fixed our problem - the key difference was we used the SagaUniqueIdentity.FormatId() to generate our document IDs rather than a new guid - this was trivial tio do since we were already referencing the NServiceBus and NServiceBus.Core assemblies.
The short answer is that it is not enough to make the data resemble the new identity documents. Where you are using Guid.NewGuid().ToString(), that data is important! That's why your solution isn't working right now. I spoke about the concept of identity documents (specifically about the NServiceBus use case) during the last quarter of my talk at RavenConf 2014 - here are the slides and video.
So here is the long answer:
In RavenDB, the only ACID guarantees are on the Load/Store by Id operations. So if two threads are acting on the same Saga concurrently, and one stores the Saga data, the second thread can only expect to get back the correct saga data if it is also loading a document by its Id.
To guarantee this, the Raven Saga Persister uses an identity document like the one you showed. It contains the SagaId, the UniqueValue (mostly for human comprehension and debugging, the database doesn't technically need it), and the SagaDocId (which is a little duplication as its only the {SagaTypeName}/{SagaId} where we already have the SagaId.
With the SagaDocId, we can use the Include feature of RavenDB to do a query like this (which is from memory, probably wrong, and should only serve to illustrate the concept as pseudocode)...
var identityDocId = // some value based on incoming message
var idDoc = RavenSession
// Look at the identity doc's SagaDocId and pull back that document too!
.Include<SagaIdentity>(identityDoc => identityDoc.SagaDocId)
.Load(identityDocId);
var sagaData = RavenSession
.Load(idDoc.SagaDocId); // Already in-memory, no 2nd round-trip to database!
So then the identityDocId is very important because it describes the uniqueness of the value coming from the message, not just any old Guid will do. So what we really need to know is how to calculate that.
For that, the NServiceBus saga persister code is instructive:
void StoreUniqueProperty(IContainSagaData saga)
{
var uniqueProperty = UniqueAttribute.GetUniqueProperty(saga);
if (!uniqueProperty.HasValue) return;
var id = SagaUniqueIdentity.FormatId(saga.GetType(), uniqueProperty.Value);
var sagaDocId = sessionFactory.Store.Conventions.FindFullDocumentKeyFromNonStringIdentifier(saga.Id, saga.GetType(), false);
Session.Store(new SagaUniqueIdentity
{
Id = id,
SagaId = saga.Id,
UniqueValue = uniqueProperty.Value.Value,
SagaDocId = sagaDocId
});
SetUniqueValueMetadata(saga, uniqueProperty.Value);
}
The important part is the SagaUniqueIdentity.FormatId method from the same file.
public static string FormatId(Type sagaType, KeyValuePair<string, object> uniqueProperty)
{
if (uniqueProperty.Value == null)
{
throw new ArgumentNullException("uniqueProperty", string.Format("Property {0} is marked with the [Unique] attribute on {1} but contains a null value. Please make sure that all unique properties are set on your SagaData and/or that you have marked the correct properties as unique.", uniqueProperty.Key, sagaType.Name));
}
var value = Utils.DeterministicGuid.Create(uniqueProperty.Value.ToString());
var id = string.Format("{0}/{1}/{2}", sagaType.FullName.Replace('+', '-'), uniqueProperty.Key, value);
// raven has a size limit of 255 bytes == 127 unicode chars
if (id.Length > 127)
{
// generate a guid from the hash:
var key = Utils.DeterministicGuid.Create(sagaType.FullName, uniqueProperty.Key);
id = string.Format("MoreThan127/{0}/{1}", key, value);
}
return id;
}
This relies on Utils.DeterministicGuid.Create(params object[] data) which creates a Guid out of an MD5 hash. (MD5 sucks for actual security but we are only looking for likely uniqueness.)
static class DeterministicGuid
{
public static Guid Create(params object[] data)
{
// use MD5 hash to get a 16-byte hash of the string
using (var provider = new MD5CryptoServiceProvider())
{
var inputBytes = Encoding.Default.GetBytes(String.Concat(data));
var hashBytes = provider.ComputeHash(inputBytes);
// generate a guid from the hash:
return new Guid(hashBytes);
}
}
}
That's what you need to replicate to get your utility to work properly.
What's really interesting is that this code made it all the way to production - I'm surprised you didn't run into trouble before this, with messages creating new saga instances when they really shouldn't because they couldn't find the existing Saga data.
I almost think it might be a good idea if NServiceBus would raise a warning any time you tried to find Saga Data by anything other than a [Unique] marked property, because it's an easy thing to forget to do. I filed this issue on GitHub and submitted this pull request to do just that.
We have an entity named Organization that we use the UniqueConstraints-bundle on. We have a property named NetName that is a UniqueConstraint and an automaticly generated Id.
Since this is unneccesary we want to use the NetName-property as Id instead. So that we don't need UniqueConstraints to know that it is unique and also get the benefit from being able to use Load when we have the NetName.
We needed to clean up our netname a bit before using it as an Id so we created a new temporary-property called TempUniqueNetName that now holds the value of:
"organizations/"+ CleanupId(this.NetName)
So we are now ready to simply move that value to our Id. But we can't get it to work. Our problem is that with the PatchRequest below we end up with a new property named Id in the database but the acctual Id still has the same value (see screenshot). Is there a better (correct) way to change the value of an Id?
The Entity:
class Organization {
public string Id { get; set; }
[UniqueConstraint]
public string NetName { get; set; }
public string TempUniqueNetName{ get; set; }
}
We want to do something like this:
_documentStore.DatabaseCommands.UpdateByIndex(typeof(Organizations).Name,
new IndexQuery(),
new[]
{
new PatchRequest()
{
Type = PatchCommandType.Rename,
Name = "TempUniqueNetName",
Value = new RavenJValue("Id")
}
});
I don't think you can change the document key via patching. It's not actually stored with the document or the metadata - it's copied into the #id metadata on load to give you the illusion that it's there, and the Raven Client copies it again into your own identity property in the document. But really, it's a separate value in the underlying esent document store. Raven would have to know specifically how to handle this and fake it for you.
You could manually copy the doc from the old id to the new one and delete the old, but that could be time consuming.
There isn't a great answer for renaming a document key right now. There really should be a DatabaseCommand for rekeying a single document, and separate PatchCommandType to rekey when patching. Perhaps this will be added to raven in the future.
You can check implemtation of PUT-DELETE usage for updating IDs in my github repo.
It should look something like this:
store.DatabaseCommands.Put(updatedKey, null, document.DataAsJson, newMetadata);
store.DatabaseCommands.Delete(oldKey, null);
https://github.com/Sevsoad/SagaUpdater/
Also here is some Raven documentation:
https://ravendb.net/docs/article-page/3.0/csharp/client-api/commands/documents/put
I need to write a row to the database regardless of whether it already exists or not. Before using NHibernate this was done with a stored procedure. The procedure would attempt an update and if no rows were modified it would fallback to an insert. This worked well because the application doesn't care if the record exists.
With NHibernate, the solutions I have found require loading the entity and modifying it, or deleting the entity so the new one can be inserted. The application does have to care if the record already exists. Is there a way around that?
Does the Id Matter?
Assigned Id
The object has a keyword as an assigned id and is the primary key in the table.
I understand that SaveOrUpdate() will call the Save() or Update() method as appropriate based on the Id. Using an assigned id, this won't work because the id isn't an unsaved-value. However a Version or Timestamp field could be used as an indicator instead. In reality, this isn't relevant because this only reflects on whether the object in memory has been associated with a record in the database; it does not indicate if the record exists or not in the database.
Generated Id
If the assigned id were truly the cause of the problem, I could use a generated id instead of the keyword as the primary key. This would avoid the NHibernate Insert/Update issue as it would effectively always insert. However, I still need to prevent duplicate keywords. With a unique index on the keyword column it will still throw an exception for a duplicate keyword even if the primary key is different.
Another Approach?
Perhaps the problem isn't really with NHibernate, but the way this is modeled. Unlike other areas of the application, this is more data-centric rather object-centric. It is nice that NHibernate makes it easy to read/write and eliminates the stored procedures. But the desire to simply write without regard to existing values doesn't fit well with the model of an object's identity model. Is there a better way to approach this?
I`m using
public IList<T> GetByExample<T>(T exampleInstance)
{
return _session.CreateCriteria(typeof(T))
.Add(Example.Create(exampleInstance))
.List<T>();
}
public void InsertOrUpdate<T>(T target)
{
ITransaction transaction = _session.BeginTransaction();
try
{
var res=GetByExample<T>(target);
if( res!=null && res.Count>0 )
_session.SaveOrUpdate(target);
else
_session.Save(target);
transaction.Commit();
}
catch (Exception)
{
transaction.Rollback();
throw;
}
finally
{
transaction.Dispose();
}
}
but FindByExample method returns all objects alike not objects with the exact ID what do you suggest ? since I have only object as parameter I don't have access to its specific ID field so I cannot use session.get(Object.class(), id);
Typically, NHibernate can rely on the unsaved-value to determine whether it should insert or create the entity. However, since you are assigning the ID, to NHibernate it looks like your entity has already been persisted. Therefore, you need to rely on versioning your object to let NHibernate know that it is a new object. See the following link for how to version your entity:
http://web.archive.org/web/20090831032934/http://devlicio.us/blogs/mike_nichols/archive/2008/07/29/when-flushing-goes-bad-assigned-ids-in-nhibernate.aspx
Use the session.SaveOrUpdate(object) method.
You can do
Obj j = session.get(Object.class(), id);
if (j != null)
session.merge(myObj);
else
session.saveOrUpdate(myObj);
Query objects where keyword = x, take FirstOrDefault. If it's null, Add new object, if it exists, update object that you got and call saveOrUpdate on it.
This worked for me:
Implementation
public void InsertOrUpdate<TEntity, TId>(TEntity entity) where TEntity : IIdentificableNh<TId>
{
var anyy = session.Get<TEntity>(entity.Id);
if (anyy != null)
{
session.Evict(anyy); //dispatch all data loaded, to allow updating 'entity' object.
session.Update(entity);
}
else
{
session.Save(entity);
}
session.Flush();
}
Entity
public class Caracteristica : IIdentificableNh<int>
{
public virtual int Id { get; set; }
public virtual string Descripcion { get; set; }
}
I had to create an interface (IIdentificableNh) that allows me to access the Id property value.
Usage example:
session.InsertOrUpdate<Caracteristica, int>(new Caracteristica { Id = 2, Descripcion = "Caracteristica2" });
call hibernate.saveOrUpdate() which will check if the object is in the database, update it if it is, and save (i.e. insert) it if it is not.