rI've looked at the questions and indeed the RavenDb docs. There's a little at RavenDb Index Replication Docs but there doesn't seem any guidance on how/when/where to create the IndexReplicationDestination
Our use case is very simple (it's a spike). We currently create new objects (Cows) and store them in Raven. We have a couple of queries created dynamically using LINQ (e.g. from c in session.Query<Cows> select c).
Now I can't see where I should define the index to replicate. Any ideas? I've got hold of the bundle and added it to the server directory (I'm assuming it should be in RavenDB.1.0.499\server\Plugins where RavenDB.1.0.499\server contains Raven.Server.exe)
Edit: Thanks Ayende... the answer below and in the ravendb groups helped. There was a facepalm moment. Regardless here's some detail that may help someone else. It really is very easy and indeed 'just works':
a) Ensure that the plugins are being picked up. You can view these in the statistics - available via the /localhost:8080/stats url (assuming default settings). You should see entries in 'Extensions' regarding to the IndexReplication bundle.
If not present ensure the versions of the DLLs (bundle and server) are the same
b) Ensure the index you want to replicate has been created. They can be created via Client API or HTTP API.
Client API:
public class Cows_List : AbstractIndexCreationTask<Cow>
{
public Cows_List()
{
Map = cows => from c in cows select new { c.Status };
Index( x => x.Status, FieldIndexing.Analyzed);
}
}
HTTP API (in studio):
//Cows/List
docs.Cows
.Select(q => new {Status = q.Status})
c) create the replication document. The clue here is DOCUMENT. Like everything stored, it too is a document. So after creating it must be stored in the Db :
var replicationDocument = new Raven.Bundles.IndexReplication.Data.IndexReplicationDestination
{
Id = "Raven/IndexReplication/Cows_List", ColumnsMapping = { {"Status", "Status"} },
ConnectionStringName = "Reports", PrimaryKeyColumnName = "Id",
TableName = "cowSummaries"
};
session.Store(replicationDocument);
sesson.SaveChanges();
d) Ensure you have the following in the CLIENT (e.g. MVC app or Console)
e) Create the RDBMS schema. I have a table in 'cowReports' :
CREATE TABLE [dbo].[cowSummaries](
[Id] nvarchar NULL,
[Status] nchar NULL)
My particular problem was not adding the index document to the store. I know. facepalm. Of course everything is a document. Works like a charm!
You need to define two things.
a) An index that transform the document to the row shape.
b) A document that tell RavenDB what is the connection string name, the table name, and the columns to map
Related
I'm studying Sensenet Framework and installed successfull on my computer, and now I'm developing our website based on this framework.I read documents on wiki and understood relationship between Database <-> Properties <--> Fields <-> View (you can see the image in this link: http://wiki.sensenet.com/Field_-_for_Developers). For suppose, if I added a new table in to Sensenet's database and desiderate show all datas inside this table to our page, but I don't know how to dev flow by this model: Database <=> Property <=> Field <=> View. ? can you show steps to help me?
Please consider storing your data in the SenseNet Content Repository instead of keeping custom tables in the database. It is much easier to work with regular content items and you will have all the feature the repo offers - e.g. indexing, permissions, and of course an existing UI. To do this, you will have to take the following steps:
Define content types in SenseNet for every entity type you have in your existing db (in the example below this is the Car type).
Create a container in the Content Repository where you want to put your content (in this case this is a Cars custom list under the default site).
Create a command line tool using the SenseNet Client library to migrate your existing data to the Content Repository.
To see the example in detail, please check out this article:
How to migrate an existing database to the Content Repository
The core of the example is really a few lines of code that actually saves content items into the Content Repository (through the REST API):
using (var conn = new SqlConnection(ConnectionString))
{
await conn.OpenAsync();
using (var command = new SqlCommand("SELECT * FROM Cars", conn))
{
using (var reader = await command.ExecuteReaderAsync())
{
while (await reader.ReadAsync())
{
var id = reader.GetInt32(0);
var make = reader.GetString(1);
var model = reader.GetString(2);
var price = reader.GetInt32(3);
// Build a new content in memory and fill custom metadata fields. No need to create
// strongly typed objects here as the client Content is a dynamic type.
// Parent path is a Content Repository path, e.g. "/Root/Sites/Default_Site/Cars"
dynamic car = Content.CreateNew(ParentPath, "Car", "Car-" + id);
car.Make = make;
car.Model = model;
car.Price = price;
// save it through the HTTP REST API
await car.SaveAsync();
Console.WriteLine("Car-" + id + " saved.");
}
}
}
}
I am using Asp net 5, NHibernate 3.3 and Kendo UI MVC wrapper for grid to render the table of client orders. There are lots of orders in database already and the number is constantly growing. So I decided to use server side paging to avoid fetching all orders from database. As far as I know you can't do paging manually and delegate filtering, sorting and grouping to ToDataSourceResult method. It's either all or nothing. Therefore I made an attempt to implement so called 'custom binding'. No problem until I get to grouping. I need to group first, then sort inside of a group, then extract data for specific page and all that without loading all data to memory. My code is something like this (I put it all in one piece to simplify reading):
var orderList = CurrentSession.QueryOver<Order>();
// Filtering. Filter is a search string obtained from DataSourceRequest
var disjunction = new Disjunction();
disjunction.Add(Restrictions.On<Order>(e => e.Number).IsLike("%" + filter + "%"));
disjunction.Add(Restrictions.On<Order>(e => e.Customer).IsLike("%" + filter + "%"));
orderList = orderList.Where(disjunction);
// Sorting. sortColumn is also from DataSourceRequest
switch (sortColumn)
{
case "Number":
orderList = orderList.OrderBy(x => x.Number).Desc;
break;
case "GeneralInfo.LastChangeDate":
orderList = orderList.OrderBy(x => x.LastChangeDate).Desc;
break;
default:
orderList = orderList.OrderBy(x => x.Number).Desc;
break;
}
}
// Total is required for kendo grid when you do paging manually
var total = orderList.RowCount();
var orders = orderList
.Fetch(x => x.OrderGoods).Eager
.Fetch(x => x.OrderComments).Eager
.Fetch(x => x.Documents).Eager
.Fetch(x => x.Partner).Eager
.Skip((request.Page - 1)*request.PageSize).Take(request.PageSize).List();
I will be glad to have any advice on how to add grouping here.
I worked for literally months to figure out server-side grouping using the Kendo DataSource with a Kendo Grid. Paging, sorting and filtering were fairly easy. But for whatever reason, Telerik did not offer sufficient support documentation for such a critical LOB process as grouping. I’m glad you posted this question so I’d have an opportunity to share my code.
The Solution
Basically, the solution comes down to knowing 2 key parts, and they can be viewed in the following sample project: https://www.dropbox.com/s/ygtk8rwl1hwjvth/KendoServerGrouping.zip?dl=0
There is a single web application project in the Visual Studio (2012 | 2013) solution you’re downloading, which contains a reference to the Kendo.Mvc library. You can download the latest UI for ASP.NET binaries from Telerik's Control Panel installation program. The binaries will be located in the following Windows directory after install: C:\Program Files (x86)\Telerik\UI for ASP.NET MVC [Telerik Release Version]\wrappers\aspnetmvc\Binaries\ [Your Version of MVC]\Kendo.Mvc.dll.
Note: My solution uses Telerik’s MVC transport mechanism, which provides full-fledged server-side paging, filtering, sorting and, most notably, grouping. However, I use pure JavaScript to configure the Kendo DataSource and not the MVC wrappers. Still, I've recently found a link in Telerik's documentation that shows the MVC wrapper declaration in Razor/ASPX.
The Server Magic
Basically, the first part of the magic is the following server side code, residing in the sample WebApi controller in the KendoServerGrouping.Web\Controllers directory:
[System.Web.Http.AcceptVerbs("GET", "ASPNETMVC-AJAX")]
public Kendo.Mvc.UI.DataSourceResult GetAllAccounts([System.Web.Http.ModelBinding.ModelBinder(typeof(WebApiDataSourceRequestModelBinder))] Kendo.Mvc.UI.DataSourceRequest request)
{
var kendoRequest = new Kendo.Mvc.UI.DataSourceRequest
{
Page = request.Page,
PageSize = request.PageSize,
Filters = request.Filters,
Sorts = request.Sorts,
Groups = request.Groups,
Aggregates = request.Aggregates
};
// Set this to your ORM or other data source
IQueryable accounts = dbContext.Accounts;
/*
The data source can even be a MongoDB collection using the
.AsQueryable() extension and the MongoDB C# driver
var accounts = collection.FindAllAs<Account>().AsQueryable();
*/
var data = accounts.ToDataSourceResult(kendoRequest);
var result = new DataSourceResult()
{
AggregateResults = data.AggregateResults,
Data = data.Data,
Errors = data.Errors,
Total = data.Total
};
return result;
}
This is all you’ll need for any of the four server-side actions that the grid will handle auto-magically when the user interacts with it. Pay special attention to the AcceptVerbs attribute above the method; it must include the “ASPNETMVC-AJAX” attribute for the DataSourceRequest input parameter to work properly. ToDataSourceResult() is an extension provided by recent versions of the Kendo.Mvc.dll library, which I pointed to earlier.
The code above will (to my knowledge) work with any IQueryable data source, such as those from ORMs (I've tested Entity Framework and Telerik Data Access/Open Access). I've also been able to group a MongoDB collection using the MongoDB C# driver. However, this is meant as a proof-of-concept, and it has not been tested for performance.
For the purposes of this example, there is static data source in the WebAPI controller to fake an IQueryable collection. Naturally, you can delete the static data from lines 45-57 when you've swapped in your own data source.
The Client Magic
The Kendo DataSource automatically passes in a specialized DataSourceRequest object from the grid containing all the parameters for server-side paging, filtering, sorting and grouping, provided you wrap your DataSource schema inside the following JavaScript:
schema: $.extend(true, {}, kendo.data.schemas["aspnetmvc-ajax"], {
});
This was perhaps the single most elusive line of code I’ve ever tracked down. It took about a dozen exchanges with Telerik over several months to get them to cough it up. And even then, it was by pure chance that it was revealed. Why such a critical nuance was absent in their documentation is beyond me.
Carefully review each of the Kendo DataSource configuration settings toward the bottom half of index.html file. Most importantly, pay attention to what isn’t there, such as the batch and mvcTransport options. Including the latter option somehow negates the above “aspnetmvc-ajax” schema attribute.
In the DataSource's parmaterMap function, make note that when – and only when - performing a read operation, the following line must be present:
return mvcTransport.parameterMap(options, operation);
You will also want to be sure to include this in your HTML, before the DataSource executes:
<script src="//cdn.kendostatic.com/[Version]/js/kendo.aspnetmvc.min.js"></script>
The End Result
Run the KendoServerGrouping.Web project (index.html) and, if all goes well, a grid will be populated with 5 records containing AccountId, AccountName, AccountTypeCode and CreatedOn fields. If you set the number of visible grid rows to 2 and group by AccountTypeCode or CreatedOn, you’ll see that the grouping traverses the paging, which I believe is the end result you are looking for.
I hope the sample project works and is a good fit for your situation. Please let me know if you have any questions, and I’ll do my best to help.
P.S. This is my first post to SO, so please go easy on me if something isn’t up to SO standards. Good luck!
I would like to add onto #aaron-jessen answer with this jewel I found on Telerik's forums:
$("#grid").kendoGrid({
dataSource: {
type: "aspnetmvc-ajax", // If missing may cause NULL values in ApiController
}
})
I have a bunch of existing sagas in various states of a long running process.
Recently we decided to make one of the properties on our IContainSagaData implementation unique by using the Saga.UniqueAttribute (about which more here http://docs.particular.net/nservicebus/nservicebus-sagas-and-concurrency).
After deploying the change, we realized that all our old saga instances were not being found, and after further digging (thanks Charlie!) discovered that by adding the unique attribute, we were required to data fix all our existing sagas in Raven.
Now, this is pretty poor, kind of like adding a index to a database column and then finding that all the table data no longer select-able, but being what it is, we decided to create a tool for doing this.
So after creating and running this tool we've now patched up the old sagas so that they now resemble the new sagas (sagas created since we went live with the change).
However, despite all the data now looking right we're still not able to find old instances of the saga!
The tool we wrote does two things. For each existing saga, the tool:
Adds a new RavenJToken called "NServiceBus-UniqueValue" to the saga metadata, setting the value to the same value as our unique property for that saga, and
Creates a new document of type NServiceBus.Persistence.Raven.SagaPersister.SagaUniqueIdentity, setting the SagaId, SagaDocId, and UniqueValue fields accordingly.
My questions are:
Is it sufficient to simply make the data look correct or is there something else we need to do?
Another option we have is to revert the change which added the unique attribute. However in this scenario, would those new sagas which have been created since the change went in be OK with this?
Code for adding metadata token:
var policyKey = RavenJToken.FromObject(saga.PolicyKey); // This is the unique field
sagaDataMetadata.Add("NServiceBus-UniqueValue", policyKey);
Code for adding new doc:
var policyKeySagaUniqueId = new SagaUniqueIdentity
{
Id = "Matlock.Renewals.RenewalSaga.RenewalSagaData/PolicyKey/" + Guid.NewGuid().ToString(),
SagaId = saga.Id,
UniqueValue = saga.PolicyKey,
SagaDocId = "RenewalSaga/" + saga.Id.ToString()
};
session.Store(policyKeySagaUniqueId);
Any help much appreciated.
EDIT
Thanks to David's help on this we have fixed our problem - the key difference was we used the SagaUniqueIdentity.FormatId() to generate our document IDs rather than a new guid - this was trivial tio do since we were already referencing the NServiceBus and NServiceBus.Core assemblies.
The short answer is that it is not enough to make the data resemble the new identity documents. Where you are using Guid.NewGuid().ToString(), that data is important! That's why your solution isn't working right now. I spoke about the concept of identity documents (specifically about the NServiceBus use case) during the last quarter of my talk at RavenConf 2014 - here are the slides and video.
So here is the long answer:
In RavenDB, the only ACID guarantees are on the Load/Store by Id operations. So if two threads are acting on the same Saga concurrently, and one stores the Saga data, the second thread can only expect to get back the correct saga data if it is also loading a document by its Id.
To guarantee this, the Raven Saga Persister uses an identity document like the one you showed. It contains the SagaId, the UniqueValue (mostly for human comprehension and debugging, the database doesn't technically need it), and the SagaDocId (which is a little duplication as its only the {SagaTypeName}/{SagaId} where we already have the SagaId.
With the SagaDocId, we can use the Include feature of RavenDB to do a query like this (which is from memory, probably wrong, and should only serve to illustrate the concept as pseudocode)...
var identityDocId = // some value based on incoming message
var idDoc = RavenSession
// Look at the identity doc's SagaDocId and pull back that document too!
.Include<SagaIdentity>(identityDoc => identityDoc.SagaDocId)
.Load(identityDocId);
var sagaData = RavenSession
.Load(idDoc.SagaDocId); // Already in-memory, no 2nd round-trip to database!
So then the identityDocId is very important because it describes the uniqueness of the value coming from the message, not just any old Guid will do. So what we really need to know is how to calculate that.
For that, the NServiceBus saga persister code is instructive:
void StoreUniqueProperty(IContainSagaData saga)
{
var uniqueProperty = UniqueAttribute.GetUniqueProperty(saga);
if (!uniqueProperty.HasValue) return;
var id = SagaUniqueIdentity.FormatId(saga.GetType(), uniqueProperty.Value);
var sagaDocId = sessionFactory.Store.Conventions.FindFullDocumentKeyFromNonStringIdentifier(saga.Id, saga.GetType(), false);
Session.Store(new SagaUniqueIdentity
{
Id = id,
SagaId = saga.Id,
UniqueValue = uniqueProperty.Value.Value,
SagaDocId = sagaDocId
});
SetUniqueValueMetadata(saga, uniqueProperty.Value);
}
The important part is the SagaUniqueIdentity.FormatId method from the same file.
public static string FormatId(Type sagaType, KeyValuePair<string, object> uniqueProperty)
{
if (uniqueProperty.Value == null)
{
throw new ArgumentNullException("uniqueProperty", string.Format("Property {0} is marked with the [Unique] attribute on {1} but contains a null value. Please make sure that all unique properties are set on your SagaData and/or that you have marked the correct properties as unique.", uniqueProperty.Key, sagaType.Name));
}
var value = Utils.DeterministicGuid.Create(uniqueProperty.Value.ToString());
var id = string.Format("{0}/{1}/{2}", sagaType.FullName.Replace('+', '-'), uniqueProperty.Key, value);
// raven has a size limit of 255 bytes == 127 unicode chars
if (id.Length > 127)
{
// generate a guid from the hash:
var key = Utils.DeterministicGuid.Create(sagaType.FullName, uniqueProperty.Key);
id = string.Format("MoreThan127/{0}/{1}", key, value);
}
return id;
}
This relies on Utils.DeterministicGuid.Create(params object[] data) which creates a Guid out of an MD5 hash. (MD5 sucks for actual security but we are only looking for likely uniqueness.)
static class DeterministicGuid
{
public static Guid Create(params object[] data)
{
// use MD5 hash to get a 16-byte hash of the string
using (var provider = new MD5CryptoServiceProvider())
{
var inputBytes = Encoding.Default.GetBytes(String.Concat(data));
var hashBytes = provider.ComputeHash(inputBytes);
// generate a guid from the hash:
return new Guid(hashBytes);
}
}
}
That's what you need to replicate to get your utility to work properly.
What's really interesting is that this code made it all the way to production - I'm surprised you didn't run into trouble before this, with messages creating new saga instances when they really shouldn't because they couldn't find the existing Saga data.
I almost think it might be a good idea if NServiceBus would raise a warning any time you tried to find Saga Data by anything other than a [Unique] marked property, because it's an easy thing to forget to do. I filed this issue on GitHub and submitted this pull request to do just that.
From RavenDB's documentation:
The indexes each RavenDB server instance uses to facilitate fast
queries are powered by Lucene, the full-text search engine.
Lucene takes a Document , breaks it down into Fields , and then splits
all text in a Field into tokens ( Terms ) in a process called
Tokenization . Those tokens are what will be stored in the index, and
be later searched upon.
{...}
After the tokenization and analysis process is complete, the resulting
tokens are stored in an index, which is now ready to be search with.
{...}
The default values for each field are FieldStorage.No in Stores and
FieldIndexing.Default in Indexes.
Setting FieldIndexing.No causes values to not be available in where
clauses when querying (similarly to not being present in the original
projection). FieldIndexing.NotAnalyzed causes whole properties to be
treated as a single token and matches must be exact, similarly to
using a KeywordAnalyzer on this field. The latter is useful for
product Ids, for example. FieldIndexing.Analyzed allows to perform
full text search operations against the field. FieldIndexing.Default
will index the field as a single term, in lower case.
As I understand it, to create a RavenDB index, you simply need to specify the Map propertly, like the following:
public class PlayersIndex : AbstractIndexCreationTask<Player>
{
public PlayersIndex()
{
Map = players => from doc in players
select new { doc.PlayerId, doc.TeamId, doc.PositionId };
}
}
Here is my question:
If you assume that PlayerId is a Guid, TeamId is an int, and PositionId is an enum, should I:
Refrain from specifying any indexing options?
Configure each field as FieldIndexing.NotAnalyzed?
In other words, should I entertain the idea of specifying the fields like the following?
public class PlayersIndex : AbstractIndexCreationTask<Player>
{
public PlayersIndex()
{
Map = players => from doc in players
select new { doc.PlayerId, doc.TeamId, doc.PositionId };
Indexes.Add(x => x.PlayerId, FieldIndexing.NotAnalyzed);
Indexes.Add(x => x.TeamId, FieldIndexing.NotAnalyzed);
Indexes.Add(x => x.PositionId, FieldIndexing.NotAnalyzed);
}
}
Jim,
For your needs, you aren't going to have to specify any indexing options.
I have a document in RavenDB that looks looks like:
{
"ItemId": 1,
"Title": "Villa
}
With the following metadata:
Raven-Clr-Type: MyNamespace.Item, MyNamespace
Raven-Entity-Name: Doelkaarten
So I serialized with a type MyNamespace.Item, but gave it my own Raven-Entity-Name, so it get its own collection.
In my code I define an index:
public class DoelkaartenIndex : AbstractIndexCreationTask<Item>
{
public DoelkaartenIndex()
{
// MetadataFor(doc)["Raven-Entity-Name"].ToString() == "Doelkaarten"
Map = items => from item in items
where MetadataFor(item)["Raven-Entity-Name"].ToString() == "Doelkaarten"
select new {Id = item.ItemId, Name = item.Title};
}
}
In the Index it is translated in the "Maps" field to:
docs.Items
.Where(item => item["#metadata"]["Raven-Entity-Name"].ToString() == "Doelkaarten")
.Select(item => new {Id = item.ItemId, Name = item.Title})
A query on the index never gives results.
If the Maps field is manually changed to the code below it works...
from doc in docs
where doc["#metadata"]["Raven-Entity-Name"] == "Doelkaarten"
select new { Id = doc.ItemId, Name=doc.Title };
How is it possible to define in code the index that gives the required result?
RavenDB used: RavenHQ, Build #961
UPDATE:
What I'm doing is the following: I want to use SharePoint as a CMS, and use RavenDB as a ready-only replication of the SharePoint list data. I created a tool to sync from SharePoint lists to RavenDB. I have a generic type Item that I create from a SharePoint list item and that I serialize into RavenDB. So all my docs are of type Item. But they come from different lists with different properties, so I want to be able to differentiate. You propose to differentiate on an additional property, this would perfectly work. But then I will see all list items from all lists in one big Items collection... What would you think to be the best approach to this problem? Or just live with it? I want to use the indexes to create projections from all data in an Item to the actual data that I need.
You can't easily change the name of a collection this way. The server-side will use the Raven-Entity-Name metadata, but the client side will determine the collection name via the conventions registered with the document store. The default convention being to use the type name of the entity.
You can provide your own custom convention by assigning a new function to DocumentStore.Conventions.FindTypeTagName - but it would probably be cumbersome to do that for every entity. You could create a custom attribute to apply to your entities and then write the function to look for and understand that attribute.
Really the simplest way is just to call your entity Doelkaarten instead of Item.
Regarding why the change in indexing works - it's not because of the switch in linq syntax. It's because you said from doc in docs instead of from doc in docs.Items. You probably could have done from doc in docs.Doelkaartens instead of using the where clause. They are equivalent. See this page in the docs for further examples.