RavenDB static index on dictionary - ravendb

I have an application that uses documents, that contain list of attributes in a dictionary, for some reason we need to use a static index and query/filter over these attributes.
A prototype looks like this:
class Program
{
static void Main(string[] args)
{
IDocumentStore store = new DocumentStore() { DefaultDatabase = "Test", Url = "http://localhost:8081" };
store.Initialize();
IndexCreation.CreateIndexes(typeof(Program).Assembly, store);
using (var session = store.OpenSession())
{
session.Store(new Document { Id = "1", Name = "doc_name", Attributes = new Dictionary<string, object> { { "Type", "1" }, { "Status", "Active" } } });
session.SaveChanges();
}
using (var session = store.OpenSession())
{
// works
var l1 = session.Query<Document, Documents_Index>().Where(a => a.Attributes["Type"] == "1").ToList();
// not working
var l2 = session.Query<Document, Documents_Index>().Where(a => a.Attributes["Status"] == "Active").ToList();
}
}
}
public class Documents_Index : AbstractIndexCreationTask<Document>
{
public Documents_Index()
{
Map = docs => docs.Select(a =>
new
{
a.Name,
a.Attributes,
Attributes_Type = a.Attributes["Type"]
});
}
}
[Serializable]
public class Document
{
public string Id { get; set; }
public string Name { get; set; }
public Dictionary<string, object> Attributes { get; set; }
}
But since I need to query using any arbitrary Attribute name/value this index does solve our problem. Actually the list of attributes is known at run-time (so we tried modifying the Map expression to inject any number of attribute names, but so far we weren't successful). Is there a way how to define the index in some dynamic fashion?

You need to write it like:
public class Documents_Index : AbstractIndexCreationTask<Document>
{
public Documents_Index()
{
Map = docs => docs.Select(a =>
new
{
a.Name,
_ = a.Attributes.Select(x=>CreateField("Attributes_"+x.Key, x.Value),
});
}
}

Related

Elastisearch.net property opt in?

I'm using Elastisearch.NET with NEST 2.3. I want to use attribute mapping but I only want to index certain properties. As I understand it all properties are indexed unless you ignore them using for example [String(Ignore = true)] Is it possible to ignore all properties by default and only index the ones that have a nest attribute attached to them? Like JSON.NETs MemberSerialization.OptIn
You could do this using a custom serializer to ignore any properties not marked with a NEST ElasticsearchPropertyAttributeBase derived attribute.
void Main()
{
var pool = new SingleNodeConnectionPool(new Uri("http://localhost:9200"));
var connectionSettings = new ConnectionSettings(
pool,
new HttpConnection(),
new SerializerFactory(s => new CustomSerializer(s)));
var client = new ElasticClient(connectionSettings);
client.CreateIndex("demo", c => c
.Mappings(m => m
.Map<Document>(mm => mm
.AutoMap()
)
)
);
}
public class Document
{
[String]
public string Field1 { get; set;}
public string Field2 { get; set; }
[Number(NumberType.Integer)]
public int Field3 { get; set; }
public int Field4 { get; set; }
}
public class CustomSerializer : JsonNetSerializer
{
public CustomSerializer(IConnectionSettingsValues settings, Action<JsonSerializerSettings, IConnectionSettingsValues> settingsModifier) : base(settings, settingsModifier) { }
public CustomSerializer(IConnectionSettingsValues settings) : base(settings) { }
public override IPropertyMapping CreatePropertyMapping(MemberInfo memberInfo)
{
// if cached before, return it
IPropertyMapping mapping;
if (Properties.TryGetValue(memberInfo.GetHashCode(), out mapping))
return mapping;
// let the base method handle any types from NEST
// or Elasticsearch.Net
if (memberInfo.DeclaringType.FullName.StartsWith("Nest.") ||
memberInfo.DeclaringType.FullName.StartsWith("Elasticsearch.Net."))
return base.CreatePropertyMapping(memberInfo);
// Determine if the member has an attribute
var attributes = memberInfo.GetCustomAttributes(true);
if (attributes == null || !attributes.Any(a => typeof(ElasticsearchPropertyAttributeBase).IsAssignableFrom(a.GetType())))
{
// set an ignore mapping
mapping = new PropertyMapping { Ignore = true };
Properties.TryAdd(memberInfo.GetHashCode(), mapping);
return mapping;
}
// Let base method handle remaining
return base.CreatePropertyMapping(memberInfo);
}
}
which produces the following request
PUT http://localhost:9200/demo?pretty=true
{
"mappings": {
"document": {
"properties": {
"field1": {
"type": "string"
},
"field3": {
"type": "integer"
}
}
}
}
}

NInject kernel GetAll returns empty

I've two projects (class library projects) which implement one interface:
The first one:
public class MailPlugin : Extensibility.IProductorPlugin
{
...
}
The second one:
public class FileSystemPlugin : Extensibility.IProductorPlugin
{
...
}
Extensibility.IProductorPlugin, is a interface of a third project:
namespace Extensibility
{
public delegate void NotifyDigitalInputs(List<Domain.DigitalInput> digital_inputs);
public interface IProductorPlugin
{
String Name { get; }
String Description { get; }
String Version { get; }
List<Domain.Channel> AvailableChannels { get; }
IEnumerable<Guid> TypeGuids { get; }
event NotifyDigitalInputs OnDigitalInputs;
}
}
In my composition root, I've created this class:
namespace UI
{
public sealed class NinjectServiceLocator
{
private static readonly Lazy<NinjectServiceLocator> lazy = new Lazy<NinjectServiceLocator>(() => new NinjectServiceLocator());
public static NinjectServiceLocator Instance { get { return lazy.Value; } }
public Ninject.IKernel Kernel { get; private set; }
private NinjectServiceLocator()
{
using (var k = this.Kernel = new Ninject.StandardKernel())
{
k.Bind(b => b.FromAssembliesMatching("*")
.SelectAllClasses()
.InheritedFrom(typeof(Extensibility.IProductorPlugin))
.BindAllInterfaces()
);
}
}
}
}
So, when I want to look for all plugins, I just perform this:
protected void initialize()
{
foreach (Extensibility.IProductorPlugin productor_plugin in NinjectServiceLocator.Instance.Kernel.GetAll(typeof(Extensibility.IProductorPlugin)))
{
using (var channel_tile = new DevExpress.XtraBars.Docking2010.Views.WindowsUI.Tile() { Group = "Plugin Channels" })
{
foreach (Domain.Channel channel in productor_plugin.AvailableChannels)
{
channel_tile.Elements.Add(new DevExpress.XtraEditors.TileItemElement() { Text = channel.Name });
channel_tile.Elements.Add(new DevExpress.XtraEditors.TileItemElement() { Text = channel.Description });
this.tileContainer1.Items.Add(channel_tile);
}
}
}
}
However, GetAll returns anything.
What am I doing wrong?
I'll appreciate a lot your help.
Thanks for all.
try removing the using() from around the Kernel instantiation. a using will dispose the object at the end of the scope, which we don't want for a kernel.
using (var k = this.Kernel = new Ninject.StandardKernel())

AsProjection() throwing exception for async query

In a simple test scenario which can be setup using the following:
public class TestObj
{
public string Id { get; set; }
public string Name { get; set; }
}
public class Summary
{
public string MyId { get; set; }
public string MyName { get; set; }
}
public class TestObjs_Summary : AbstractIndexCreationTask<TestObj, Summary>
{
public TestObjs_Summary()
{
Map = docs => docs.Select(d => new { MyId = d.Id, MyName = d.Name });
Store(x => x.MyId, FieldStorage.Yes);
Store(x => x.MyName, FieldStorage.Yes);
}
}
static IDocumentStore Setup()
{
var store = new DocumentStore() { Url="http://localhost:8080" };
store.Initialize();
IndexCreation.CreateIndexes(MethodInfo.GetCurrentMethod().DeclaringType.Assembly, store);
using (var session = store.OpenSession())
{
session.Store(new TestObj { Name = "Doc1" });
session.Store(new TestObj { Name = "Doc2" });
session.SaveChanges();
}
return store;
}
I can run a simple synchronous query against the index and get the expected results (2 rows output of type Summary):
using (var session = store.OpenSession())
{
var q = session.Query<Summary>("TestObjs/Summary").AsProjection<Summary>();
Dump("Sync:", q.ToList());
}
However, if I try the same thing using an asynchronous query:
using (var session = store.OpenAsyncSession())
{
var q = session.Query<Summary>("TestObjs/Summary").AsProjection<Summary>();
q.ToListAsync().ContinueWith(t => Dump("Async:", t.Result));
}
I get an InvalidCastException:
InnerException: System.InvalidCastException
Message=Unable to cast object of type 'TestObj' to type 'Summary'.
Source=Raven.Client.Lightweight
StackTrace:
at Raven.Client.Document.InMemoryDocumentSessionOperations.ConvertToEntity[T](String id, RavenJObject documentFound, RavenJObject metadata) in c:\Builds\RavenDB-Unstable\Raven.Client.Lightweight\Document\InMemoryDocumentSessionOperations.cs:line 416
at Raven.Client.Document.InMemoryDocumentSessionOperations.TrackEntity[T](String key, RavenJObject document, RavenJObject metadata) in c:\Builds\RavenDB-Unstable\Raven.Client.Lightweight\Document\InMemoryDocumentSessionOperations.cs:line 340
at Raven.Client.Document.SessionOperations.QueryOperation.Deserialize[T](RavenJObject result) in c:\Builds\RavenDB-Unstable\Raven.Client.Lightweight\Document\SessionOperations\QueryOperation.cs:line 130
...
I suspect this is a bug, but as a RavenDB newbie, I first wanted to rule out the possibility I have screwed something up here. Can anyone see why this code would be failing?
(Note: this was run on Build 721 and on 701 and both produce the same results)
Thanks for any assistance you can provide.
It looks like a bug. I don't see anything wrong in your code. I suggest you create an issue here: http://issues.hibernatingrhinos.com

RavenDB lazy search against Index returns uninitialized statistiscs

I am trying to run lazy queries against raven db and get the counts on total matching results. I am finding when I query against a static index, a lazy search does not initialize the statistics when the query is materialized, but otherwise it comes back all right.
Below is the test to prove this behaviour.
[TestFixture]
public class CanSearchLazily
{
private const int ServerPort = 8085;
private readonly string _serverAddress = #"http://localhost:{0}".For(ServerPort);
[Test]
public void CanGetTotalResultsFromStatisticsOnLazySearchAgainstDynamicIndex()
{
CanGetTotalResultsFromStatisticsOnLazySearchAgainstAnIndex();
}
[Test]
public void CanGetTotalResultsFromStatisticsOnLazySearchAgainstStaticIndex()
{
CanGetTotalResultsFromStatisticsOnLazySearchAgainstAnIndex("UserByFirstName");
}
private void CanGetTotalResultsFromStatisticsOnLazySearchAgainstAnIndex(string indexName = "")
{
BuilderSetup.DisablePropertyNamingFor<User, string>(x => x.Id);
var users = Builder<User>.CreateListOfSize(2000).All()
.With(x => x.FirstName = GetRandom.FirstName())
.With(x => x.LastName = GetRandom.LastName())
.Build();
using (GetNewServer())
using (var store = new DocumentStore { Url = _serverAddress }.Initialize())
{
using (var session = store.OpenSession())
{
users.ForEach(session.Store);
session.SaveChanges();
IndexCreation.CreateIndexes(typeof(UserByFirstName).Assembly, store);
session.Query<User, UserByFirstName>().Customize(x => x.WaitForNonStaleResults()).ToList();
}
using (var session = store.OpenSession())
{
var names = session.Query<User>().Select(u => u.FirstName).Distinct().Take(15).ToList();
RavenQueryStatistics stats;
var query = string.IsNullOrEmpty(indexName)
? session.Query<User>().Statistics(out stats).Where(x => x.FirstName.In(names))
: session.Query<User>(indexName).Statistics(out stats).Where(x => x.FirstName.In(names));
var results = query.Take(8).Lazily();
Assert.AreEqual(8, results.Value.ToList().Count);
Assert.AreEqual(DateTime.Now.Year, stats.IndexTimestamp.Year, "the index should have the current year on its timestamp");
Assert.IsTrue(stats.TotalResults > 0, "The stats should return total results");
}
}
}
protected RavenDbServer GetNewServer(bool initializeDocumentsByEntitiyName = true)
{
var ravenConfiguration = new RavenConfiguration
{
Port = ServerPort,
RunInMemory = true,
DataDirectory = "Data",
AnonymousUserAccessMode = AnonymousUserAccessMode.All
};
if (ravenConfiguration.RunInMemory == false)
IOExtensions.DeleteDirectory(ravenConfiguration.DataDirectory);
var ravenDbServer = new RavenDbServer(ravenConfiguration);
if (initializeDocumentsByEntitiyName)
{
using (var documentStore = new DocumentStore
{
Url = _serverAddress
}.Initialize())
{
new RavenDocumentsByEntityName().Execute(documentStore);
}
}
return ravenDbServer;
}
}
[Serializable]
public class User
{
public string Id { get; set; }
public string FirstName { get; set; }
public string LastName { get; set; }
}
public class UserByFirstName : AbstractIndexCreationTask<User>
{
public UserByFirstName()
{
Map = users => from user in users
select new {user.FirstName};
}
}

Serializing object graph using MongoDB Bson serializer

I've been playing a little with the MongoDB Bson serializer, using the following piece of code:
class Program
{
public class myValue
{
public int Id = 0;
public string Label = "";
}
public class myValueMap : Dictionary<string, myValue>
{
}
public class myProdData
{
public myValueMap Mapping { get; set; }
}
public class mySystemPosition
{
public string Text { get; set; }
public myProdData ProdData { get; set; }
}
static void Main(string[] args)
{
BsonClassMap.RegisterClassMap<mySystemPosition>();
BsonClassMap.RegisterClassMap<myProdData>();
BsonClassMap.RegisterClassMap<myValueMap>();
BsonClassMap.RegisterClassMap<myValue>();
var o = new mySystemPosition()
{
ProdData = new myProdData()
{
Mapping = new myValueMap()
{
{"123", new myValue() {Id = 1, Label = "Item1"}},
{"345", new myValue() {Id = 2, Label = "Item2"}},
}
}
};
var bson = o.ToBson();
var text = Encoding.ASCII.GetString(bson);
}
}
however I don't seem to be able to get the myProdData.Mapping serialized....
Do I need to configure the MongoDB Bson serializer in a special way, to make this work?
You no need to use BsonClassMap.RegisterClassMap if you no need custom serializtion(documentation).
All your classes will be desirialzied according to default rules.
Also i am changed your example a little bit to get it work(i've replaces myValueMap class with Dictionary):
public class myProdData
{
public Dictionary<string, myValue> Mapping { get; set; }
}
static void Main(string[] args)
{
var o = new mySystemPosition()
{
ProdData = new myProdData()
{
Mapping = new Dictionary<string, myValue>()
{
{"123", new myValue() {Id = 1, Label = "Item1"}},
{"345", new myValue() {Id = 2, Label = "Item2"}},
}
}
};
var json = o.ToJson();
Console.WriteLine(json);
Console.ReadKey();
}
Here is console output(just well formatted):
{
"Text":null,
"ProdData":{
"Mapping":{
"123":{
"_id":1,
"Label":"Item1"
},
"345":{
"_id":2,
"Label":"Item2"
}
}
}
}
You can test your serializtion using ToJson() extention method, in order to view that all correct and after that use ToBson() if need.
The problem is that myValueMap derives from Dictionary. That results in a class that the AutoMap method can't handle.
I recommend you just use the Dictionary directly, as Andrew did in his reply.
Ufortunately the myValueMap is an object that I can't easily change, however it turns out, that's pretty easy to create your own (de)serializer....
public class myValueMapSerializer : IBsonSerializer
{
public object Deserialize(Bson.IO.BsonReader bsonReader, System.Type nominalType, System.Type actualType, IBsonSerializationOptions options)
{
if (nominalType != typeof(myValueMap)) throw new ArgumentException("Cannot serialize anything but myValueMap");
var res = new myValueMap();
var ser = new DictionarySerializer<string, myValue>();
var dic = (Dictionary<string, myValue>)ser.Deserialize(bsonReader, typeof(Dictionary<string, myValue>), options);
foreach (var item in dic)
{
res.Add(item.Key, item.Value);
}
return res;
}
public object Deserialize(Bson.IO.BsonReader bsonReader, System.Type nominalType, IBsonSerializationOptions options)
{
throw new Exception("Not implemented");
}
public bool GetDocumentId(object document, out object id, out IIdGenerator idGenerator)
{
id = null;
idGenerator = null;
return false;
}
public void Serialize(Bson.IO.BsonWriter bsonWriter, Type nominalType, object value, IBsonSerializationOptions options)
{
if (nominalType != typeof(myValueMap)) throw new ArgumentException("Cannot serialize anything but myValueMap");
var ser = new DictionarySerializer<string, myValue>();
ser.Serialize(bsonWriter, typeof(DictionarySerializer<string, myValue>), value, options);
}
public void SetDocumentId(object document, object id)
{
return;
}
}