RavenDB: Deserializing nested arrays - ravendb

Suppose I have the following class to be serialized and stored as a RavenDB's document:
public class Block
{
public string Id { get; set; }
public List<List<dynamic>> data { get; set; }
}
After storing, a document like this can be seen in the database:
{
"Id": "f539770a",
"columns": [
[ 90, 91, 92, 93, 94 ],
[ "C", "D", "A", "B", "C" ]
] }
I want to execute a query to retrieve the Nth list of values inside the "Columns" field:
session.Query<Block>().Where(b => b.Id == "f539770a").Select(b =>b.columns[i]);
And I get the following Error:
{"Cannot deserialize the current JSON object (e.g. {\"name\":\"value\"}) into type 'System.Collections.Generic.List`1[System.Object]' because the type requires a JSON array (e.g. [1,2,3]) to deserialize correctly.\r\nTo fix this error either change the JSON to a JSON array (e.g. [1,2,3]) or change the deserialized type so that it is a normal .NET type (e.g. not a primitive type like integer, not a collection type like an array or List) that can be deserialized from a JSON object. JsonObjectAttribute can also be added to the type to force it to deserialize from a JSON object.\r\nPath '__document_id'."}
It seems like the query is working (the server returns http 200), but there is a client-side deserialization problem.
Am I missing something?
Thanks!!
UPDATE:
I changed the data structure as the error seems to suggest:
public class Column
{
public List<dynamic> data { get; set; }
}
public class Block
{
public string Id { get; set; }
public List<Column> columns { get; set; }
}
The stored document is like:
{
"Id": "f539770a",
"columns": [
{ "data": [ 95, 96, 97, 98, 99 ] },
{ "data": [ "D", "A", "B", "C", "D" ] }
]}
After executing this query:
session.Query<Block>().Include<Block>(b => b.columns).Where(b => b.parentFileId == dbFileDescriptor.Id).Select(b => b.columns[i])
I get no exception, but the nested array is not loaded:

Remove the Include. That's not what it's used for. The Include is for pre-loading references to other documents so you don't have to make multiple database trips. You might be making RavenDB look for Block's and their corresponding Column documents. See documentation here.
session.Query<Block>().Where(b => b.parentFileId == dbFileDescriptor.Id).Select(b => b.data[i]);
EDIT:
If you're going to look something by the Id, use Load not Query.
session.Load<Block>("f539770a").data[i];
I verified that this works with a raven unit test using RavenDB.Tests.Helpers nuget package and Shouldly for the assertion.
public class SoQuestion : RavenTestBase
{
[Fact]
public void GetBlock_Success()
{
using (var docStore = NewDocumentStore())
{
using (var session = docStore.OpenSession())
{
session.Store(new Block
{
Id = "f539770a",
data = new List<List<dynamic>>()
{
new List<dynamic>()
{
90,
91,
92,
93,
94
},
new List<dynamic>()
{
"C",
"D",
"A",
"B",
"C"
}
}
});
session.SaveChanges();
}
docStore.WaitForStaleIndexesToComplete();
//Act
var block = GetBlock(docStore);
//Assert
block.ShouldNotBe(null);
block.data.ShouldNotBeEmpty();
}
}
private Block GetBlock(IDocumentStore documentStore)
{
using (var session = documentStore.OpenSession())
{
return session.Load<Block>("f539770a");
}
}
public class Block
{
public string Id { get; set; }
public List<List<dynamic>> data { get; set; }
}
}

Related

How to find from mongo collection using comma separated string in C#

I want to fetch data from mongo using comma separated string in C#. Below is my code.
string numbers = "32,12,56,78";
List<CustomerNumbers> calling_number= new List<CDRs>();
IMongoCollection<CustomerNumbers> Collec;
calling_number= Collec.Find(x => x.customer == ID && x.numbers.Contains(numbers)).ToList();
I am new to mongo and and did't know the exact approach. while using above code I am getting records for single number. Please guide me to fix this.
TIA
Class structure
public class CustomerNumbers
{
public string numbers { get; set; }
public int customer { get; set; }
}
Ideally, you'd have modeled your numbers field in mongo as an array rather than a delimited string, this would allow you to add indexes and performance tune your queries.
However, we can make use of the aggregation pipeline for this. We'll need an extra class like the following that we'll use:
public class CustomerNumbersSplit
{
public string[] numbers { get; set; }
public int customer { get; set; }
}
We'll also need to do is split your comma-delimited string.
var numbers = "32,12,56,78";
var numbersSplit = numbers.Split(",", StringSplitOptions.RemoveEmptyEntries)
.Select(int.Parse)
.ToArray();
Once we've done that we can then write a query as the following:
var result = await collection.Aggregate()
.Match(x => x.customer == ID)
.AppendStage<CustomerNumbersSplit>(#"{ $addFields: { numbers: { $split: [""$numbers"", "","" ] } } }")
.Match(x => x.numbers.Any(y => numbersSplit.Contains(y)))
.ToListAsync();
This makes use of a $split and $addFields, this is so we can use the database engine to split the number and query them in the database engine.
If you're interested in the query that it generated:
[
{ "$match" : { "customer" : 12324 } },
{ "$addFields" : { "numbers" : { "$split" : ["$numbers", ","] } } },
{ "$match" : { "numbers" : { "$elemMatch" : { "$in" : ["32", "12", "56", "78"] } } } }
]

How to extract record from json string to display in Jquery Datatable?

I'm fetching client records from the database and after concatenating the model class properties(FirstName and LastName) I converted these records in JSON string. My goal is to display these records in Jquery Datatable. I observed at debugging, all these records are in a single string causing the error "Requested unknown parameter".
At first, I tried to return a simple JSON return method return Json(ClientList) but it did not concatenate the model class properties. Thus, I used JsonConvert.SerializeObject() to convert all the details by passing ClientList object and now scratching my head how to use this JSON String in Jquery Datatable as all the records are in this format [{record 1 details..},{record 2 details..}, and so on].
Model Class
public string FirstName { get; set; }
public string LastName { get; set; }
public string FullName
{
get
{
return FirstName + " " + LastName;
}
}
Controller
public ActionResult GetClientList()
{
var ClientList = _context.Clients.ToList();
//return Json(ClientList);
string jsonData = JsonConvert.SerializeObject(clientList);
return Json(jsonData);
}
View
<script>
$(document).ready(function () {
$("#ClientTable").DataTable({
"ajax": {
"url": "/Client/GetClientList",
"type": "GET",
"datatype": "json",
"dataSrc": ""
},
"columns":
[
{ "data": "Id" },
{ "data": "FullName" }
]
});
});
</script>
I want datatable to look like this:
Id | Name
1 | ABC XYZ

Entity Framework Core import / export tree of entities with similar foreign keys

I need to add an import/export functionality to my ASP.NET Core application.
What I would like is to take entities in one database, export these entities into one file, and then import that file into a new database.
My problem is that I have some entities that hold same foreign key. Here is a simple model illustrating what I want to do:
public class Bill
{
public List<Product> Products { get; set; }
public int Id { get; set; }
...
}
public class Product
{
public int Id { get; set; }
public int ProductCategoryId { get; set; }
public ProductCategory ProductCategory { get; set; }
...
}
public class Category
{
public int Id { get; set; }
public string Name { get; set; }
}
I want to export a bill to be imported on an other environment of my application. So if I export the bill, I will get a Json like this:
{
"id": 1,
"products" : [
{
"id" : 1,
"productCategoryId": 1,
"productCategory": {
"id" : 1,
"name" : "Category #1"
}
},
{
"id" : 2,
"productCategoryId": 1,
"productCategory": {
"id" : 1,
"name" : "Category #1"
}
},
{
"id" : 3,
"productCategoryId": 1,
"productCategory": {
"id" : 2,
"name" : "Category #2"
}
}
]
}
If I deserialize this json into entities in my new environment (ignoring Ids mapping of course), I will get three new categories (category will be duplicated for product 1 and 2) because the serializer will instanciate two categories...
So when I push it into my database, it will add 3 lines instead 2 into the Category table...
Thanks in advance for your answers.
Suppose you have a list of Category to be imported, you could firstly get the id list of these categories and then query the database to make sure which has been already stored in database. And for those already existing ones, just skip them (or update them as you like).
Since we have multiple entity types (categories,products,bills and potential BillProducts), rather than writing a new importer for each TEntity , I prefer to writing a generic Importer method to deal with any Entity type list with Generic and Reflection :
public async Task ImportBatch<TEntity,TKey>(IList<TEntity> entites)
where TEntity : class
where TKey: IEquatable<TKey>
{
var ids = entites.Select( e=> GetId<TEntity,TKey>(e));
var existingsInDatabase=this._context.Set<TEntity>()
.Where(e=> ids.Any(i => i.Equals(GetId<TEntity,TKey>(e)) ))
.ToList();
using (var transaction = this._context.Database.BeginTransaction())
{
try{
this._context.Database.ExecuteSqlCommand("SET IDENTITY_INSERT " + typeof(TEntity).Name + " ON;");
this._context.SaveChanges();
foreach (var entity in entites)
{
var e= existingsInDatabase.Find(existing => {
var k1 =GetId<TEntity,TKey>(existing);
var k2=GetId<TEntity,TKey>(entity);
return k1.Equals(k2);
});
// if not yet exists
if(e == null){
this._context.Add(entity);
}else{
// if you would like to update the old one when there're some differences
// uncomment the following line :
// this._context.Entry(e).CurrentValues.SetValues(entity);
}
}
await this._context.SaveChangesAsync();
transaction.Commit();
}
catch{
transaction.Rollback();
}
finally{
this._context.Database.ExecuteSqlCommand($"SET IDENTITY_INSERT " + typeof(TEntity).Name + " OFF;");
await this._context.SaveChangesAsync();
}
}
return;
}
Here the GetId<TEntity,TKey>(TEntity e) is a simple helper method which is used to get the key filed of e:
// use reflection to get the Id of any TEntity type
private static TKey GetId<TEntity,TKey>(TEntity e)
where TEntity : class
where TKey : IEquatable<TKey>
{
PropertyInfo pi=typeof(TEntity).GetProperty("Id");
if(pi == null) { throw new Exception($"the type {typeof(TEntity)} must has a property of `Id`"); }
TKey k = (TKey) pi.GetValue(e);
return k ;
}
To make the code more reusable, we can create an EntityImporter service to hold the method above :
public class EntityImporter{
private DbContext _context;
public EntityImporter(DbContext dbContext){
this._context = dbContext;
}
public async Task ImportBatch<TEntity,TKey>(IList<TEntity> entites)
where TEntity : class
where TKey: IEquatable<TKey>
{
// ...
}
public static TKey GetId<TEntity,TKey>(TEntity e)
where TEntity : class
where TKey : IEquatable<TKey>
{
// ...
}
}
and then register the services at startup time:
services.AddScoped<DbContext, AppDbContext>();
services.AddScoped<EntityImporter>();
Test Case :
Firstly, I'll take several categories as an example :
var categories = new ProductCategory[]{
new ProductCategory{
Id = 1,
Name="Category #1"
},
new ProductCategory{
Id = 2,
Name="Category #2"
},
new ProductCategory{
Id = 3,
Name="Category #3"
},
new ProductCategory{
Id = 2,
Name="Category #2"
},
new ProductCategory{
Id = 1,
Name="Category #1"
},
};
await this._importer.ImportBatch<ProductCategory,int>(categories);
The expected results should be only 3 rows imported :
1 category #1
2 category #2
3 category #3
And here's a screenshot it works:
Finally, for your bills json, you can do as below to import the entites :
var categories = bill.Products.Select(p=>p.ProductCategory).ToList();
var products = bill.Products.ToList();
// other List<TEntity>...
// import the categories firstly , since they might be referenced by other entity
await this._importer.ImportBatch<ProductCategory,int>(categories);
// import the product secondly , since they might be referenced by BillProduct table
await this._import.ImportBatch<Product,int>(products);
// ... import others

Returning sub-properties from document in RavenDb via Index using Lucene

I'm using RavenDB 2.5 and what I want to do is query a Group (see below) providing a valid Lucene search term and get back a collection of Member instances (or even just Ids) that match. So, class definition:
public class Group {
public string Id { get; set; }
public IList<Member> Members { get; set; }
}
public class Member {
public string Name { get; set; }
public string Bio { get; set; }
}
And they are stored in the database as session.Store(groupInstance); as you'd expect. What I'd like to do is query and return the Member instances which match a given search term.
So, something like:
public class GroupMembers_BySearchTerm : AbstractIndexCreationTask {
public override IndexDefinition CreateIndexDefinition(){
return new IndexDefinition {
Map = "from g in docs.Groups select new { Content = new [] { g.Members.Select(m => m.Name), g.Members.Select(m => m.Bio) }, Id = g.Id }",
Indexes = { { "Id", FieldIndexing.Default }, { "Content", FieldIndexing.Analyzed } }
}
}
}
If I call this using something like:
session.Advanced.LuceneQuery<Group, GroupMembers_BySearchTerm>().Where("Id: myId AND Content: search terms").ToList();
I obviously get back a Group instance, but how can I get back the Members instead?
What about an index like this:
public class Members_BySearchTermAndGroup : AbstractIndexCreationTask {
public override IndexDefinition CreateIndexDefinition(){
return new IndexDefinition {
Map = "from g in docs.Groups
from member in g.Members
select new {
GroupdId = g.Id,
Name = member.Name,
Bio = member.Bio,
Content = new [] {member.Name, member.Bio },
}",
Indexes = {
{ "GroupId", FieldIndexing.Default },
{ "Content", FieldIndexing.Analyzed }
},
Stores = {
{ "Name", FieldStorage.Yes },
{ "Bio", FieldStorage.Yes }
}
}
}
}
If you take a closer look you'll see that we are creating a new lucene entry for each member inside of a group. Consequently, you'll be able to query on those elements and retrieve them.
Finally you can query your store like this (more info about searching):
session.Query<Member, Members_BySearchTermAndGroup>()
.Search(x => x.Content, "search terms")
.ProjectFromIndexFieldsInto<Member>()
.ToList();
I cannot check this right now but I guess that you need to project your results using the ProjectFromIndexFieldsInto(). Some more information about projections in this link.
or, following your example:
session.Advanced
.LuceneQuery<Member, Members_BySearchTermAndGroup>()
.Where("GroupId: myId AND Content: search terms")
.SelectFields<Member>()
.ToList();

ADO.NET Entity Framework Hierarchical Data

Consider following database model:
And following query code:
using (var context = new DatabaseEntities())
{
return context.Feed.ToHierarchy(f => f.Id_Parent, null);
}
Where ToHierarchy is an extension to ObjectSet<TEntity> as:
public static List<TEntity> ToHierarchy<TEntity, TProperty>(this ObjectSet<TEntity> set, Func<TEntity, TProperty> parentIdProperty, TProperty idRoot) where TEntity : class
{
return set.ToList().Where(f => parentIdProperty(f).Equals(idRoot)).ToList();
}
This would result in example JSON formatted response:
[
{
"Description":"...",
"Details":[ ],
"Id":1,
"Id_Parent":null,
"Title":"...",
"URL":"..."
},
{
"Description":"...",
"Details":[
{
"Description":"...",
"Details":[ ],
"Id":4,
"Id_Parent":3,
"Title":"...",
"URL":"..."
},
{
"Description":"...",
"Details":[
{
"Description":"...",
"Details":[
{
"Description":"...",
"Details":[ ],
"Id":7,
"Id_Parent":6,
"Title":"...",
"URL":"..."
}
],
"Id":6,
"Id_Parent":5,
"Title":"...",
"URL":"..."
}
],
"Id":5,
"Id_Parent":3,
"Title":"...",
"URL":null
}
],
"Id":3,
"Id_Parent":null,
"Title":"...",
"URL":null
}
]
As you may have noticed ToHierarchy method is supposed to (and apparently do indeed) retrieve all rows from a given set (flat) and return hierarchical representation of these as per "parent property".
When I was in the middle of my implementation I quick tried my code and surprisingly it worked! Now, I imagine how weird does this sound to many of you, but I really don't understand why or how that piece of code works, even though I kinda wrote it down on my own...
Could you explain how does it work?
P.S.: if you look closer, ToHierarchy is not near the same as .Include("Details").
It works because set.ToList will load all records from the database table to your application and the rest is done be EF and its change tracking mechanism which should ensure correct referencing between related entities.
Btw. you are filtering records in the memory of your application, not in the database. For example if your table contains 10000 records and your filter should return only 10, you will still load all 10000 from the database.
You will find that implementing this with EF is quite hard because EF has no support for hierarchical data. You will always end with bad solution. The only good solution is using stored procedure and some support for hierarchical queries in the database - for example common table expressions (CTE) in SQL server.
I just made this very simple example and it works as I described in comment:
public class SelfReferencing
{
public int Id { get; set; }
public string Name { get; set; }
public SelfReferencing Parent { get; set; }
public ICollection<SelfReferencing> Children { get; set; }
}
public class Context : DbContext
{
public DbSet<SelfReferencing> SelfReferencings { get; set; }
}
public class Program
{
static void Main(string[] args)
{
using (var context = new Context())
{
context.Database.Delete();
context.Database.CreateIfNotExists();
context.SelfReferencings.Add(
new SelfReferencing()
{
Name = "A",
Children = new List<SelfReferencing>()
{
new SelfReferencing()
{
Name = "AA",
Children = new List<SelfReferencing>()
{
new SelfReferencing()
{
Name = "AAA"
}
}
},
new SelfReferencing()
{
Name = "AB",
Children = new List<SelfReferencing>()
{
new SelfReferencing()
{
Name = "ABA"
}
}
}
}
});
context.SaveChanges();
}
using (var context = new Context())
{
context.Configuration.LazyLoadingEnabled = false;
context.Configuration.ProxyCreationEnabled = false;
var data = context.SelfReferencings.ToList();
}
}
}
It uses code first approach but internally it is same as when using EDMX. When I get data I have 5 entities in list and all have correctly configured Parent and Children navigation properties.