Breeze - Writing complex query with multiple parameters from client - sql

I am trying to write a Breeze query for the Northwind database to show orders that contain all three products that a user specifies. So if a user selects ProductID 41, 51, and 65 from drop downs, the query would return order id 10250.
This is just a sample scenario that I am looking to base another query on in a project I am working on, but I thought using Northwind to explain it would be easier than describing the project. I can easily do it in T-SQL using derived tables, but I need to get the parameters from the client. Any thoughts? Thanks in advance!

If you're still interested in doing this on the client, you can try the following Breeze query.
var listOfProductIds = [41, 51, 65];
var preds = listOfProductIds.map(function (id) {
return Predicate.create("OrderDetails", "any", "ProductID", "==", id);
});
var whereClause = Predicate.and(preds);
var query = EntityQuery.from('Orders').where(whereClause);
This will retrieve all Orders that have at least all 3 of the Products specified.
To further filter this so you have all Orders that have only all 3 of the Products specified, you can write,
entityManager.executeQuery(query)
.then(filterOrders);
//once you get results on the client
function filterOrders(data) {
var allOrders = data.results;
var filteredOrders = allOrders.filter(function (o) {
return o.OrderDetails.length == listOfProductIds.length;
});
}
You can only filter on the client since OData doesn't yet support the Aggregate Count function like Linq to Entities does. This is probably not ideal but it's an option if you decide not to do it on the server.

Sorry in my comment I said look at the and() method but in actuality for your use you needed to look at the or() method -
var p1 = breeze.Predicate("Product.ProductID", "==", 41);
var p2 = breeze.Predicate("Product.ProductID", "==", 51);
var p3 = breeze.Predicate("Product.ProductID", "==", 65);
var newPred = breeze.Predicate.or(p1, p2, p3);
var query = EntityQuery.from('Order_Details').select('Order.OrderID').where(newPred);
The only issue with these queries on the client is that depending on how you are building them client-side and how many predicates you are adding they can get very long in some situations, such as select all 200 records except 1 id type queries which can bite you on IE8.

You can use the EntityQuery.withParameters method to call a server side query that has your linq query. i.e. something like this.
// On the client
var query = EntityQuery.from("GetOrdersWithProductIds")
.withParameters({ productIds: [41, 51, 65] });
// On the server
[HttpGet]
public IQueryable<Order> GetOrdersWithProductIds([FromUri] int[] productIds) {
return ContextProvider.Context.Orders.Where(... your linq query here ...);
}
or you might try using the support for 'any'/'all' query operators. i.e. something like this.
var predicate = breeze.Predicate("ProductID", "==", 41)
.or("ProductID", "==", 51)
.or("ProductID", "==", 65);
var query = EntityQuery.from("Orders")
.where("OrderDetails", "all", predicate)

Related

Query Performance for multiple IQueryable in .NET Core with LINQ

I am currently updating a BackEnd project to .NET Core and having performance issues with my Linq queries.
Main Queries:
var queryTexts = from text in _repositoryContext.Text
where text.KeyName.StartsWith("ApplicationSettings.")
where text.Sprache.Equals("*")
select text;
var queryDescriptions = from text in queryTexts
where text.KeyName.EndsWith("$Descr")
select text;
var queryNames = from text in queryTexts
where !(text.KeyName.EndsWith("$Descr"))
select text;
var queryDefaults = from defaults in _repositoryContext.ApplicationSettingsDefaults
where defaults.Value != "*"
select defaults;
After getting these IQueryables I run a foreach loop in another context to build my DTO model:
foreach (ApplicationSettings appl in _repositoryContext.ApplicationSettings)
{
var applDefaults = queryDefaults.Where(c => c.KeyName.Equals(appl.KeyName)).ToArray();
description = queryDescriptions.Where(d => d.KeyName.Equals("ApplicationSettings." + appl.KeyName + ".$Descr"))
.FirstOrDefault()?
.Text1 ?? "";
var name = queryNames.Where(n => n.KeyName.Equals("ApplicationSettings." + appl.KeyName)).FirstOrDefault()?.Text1 ?? "";
// Do some stuff with data and return DTO Model
}
In my old Project, this part had an execution from about 0,45 sec, by now I have about 5-6 sec..
I thought about using compiled queries but I recognized these don't support returning IEnumerable yet. Also I tried to avoid Contains() method. But it didn't improve performance anyway.
Could you take short look on my queries and maybe refactor or give some hints how to make one of the queries faster?
It is to note that _repositoryContext.Text has compared to other contexts the most entries (about 50 000), because of translations.
queryNames, queryDefaults, and queryDescriptions are all queries not collections. And you are running them in a loop. Try loading them outside of the loop.
eg: load queryNames to a dictionary:
var queryNames = from text in queryTexts
where !(text.KeyName.EndsWith("$Descr"))
select text;
var queryNamesByName = queryName.ToDictionary(n => n.KeyName);
one can write queries like below
var Profile="developer";
var LstUserName = alreadyUsed.Where(x => x.Profile==Profile).ToList();
you can also use "foreach" like below
lstUserNames.ForEach(x=>
{
//do your stuff
});

Entity Framework filter data by string sql

I am storing some filter data in my table. Let me make it more clear: I want to store some where clauses and their values in a database and use them when I want to retrieve data from a database.
For example, consider a people table (entity set) and some filters on it in another table:
"age" , "> 70"
"gender" , "= male"
Now when I retrieve data from the people table I want to get these filters to filter my data.
I know I can generate a SQL query as a string and execute that but is there any other better way in EF, LINQ?
One solution is to use Dynamic Linq Library , using this library you can have:
filterTable = //some code to retrive it
var whereClause = string.Join(" AND ", filterTable.Select(x=> x.Left + x.Right));
var result = context.People.Where(whereClause).ToList();
Assuming that filter table has columns Left and Right and you want to join filters by AND.
My suggestion is to include more details in the filter table, for example separate the operators from operands and add a column that determines the join is And or OR and a column that determines the other row which joins this one. You need a tree structure if you want to handle more complex queries like (A and B)Or(C and D).
Another solution is to build expression tree from filter table. Here is a simple example:
var arg = Expression.Parameter(typeof(People));
Expression whereClause;
for(var row in filterTable)
{
Expression rowClause;
var left = Expression.PropertyOrField(arg, row.PropertyName);
//here a type cast is needed for example
//var right = Expression.Constant(int.Parse(row.Right));
var right = Expression.Constant(row.Right, left.Member.MemberType);
switch(row.Operator)
{
case "=":
rowClause = Expression.Equal(left, right);
break;
case ">":
rowClause = Expression.GreaterThan(left, right);
break;
case ">=":
rowClause = Expression.GreaterThanOrEqual(left, right);
break;
}
if(whereClause == null)
{
whereClause = rowClause;
}
else
{
whereClause = Expression.AndAlso(whereClause, rowClause);
}
}
var lambda = Expression.Lambda<Func<People, bool>>(whereClause, arg);
context.People.Where(lambda);
this is very simplified example, you should do many validations type casting and ... in order to make this works for all kind of queries.
This is an interesting question. First off, make sure you're honest with yourself: you are creating a new query language, and this is not a trivial task (however trivial your expressions may seem).
If you're certain you're not underestimating the task, then you'll want to look at LINQ expression trees (reference documentation).
Unfortunately, it's quite a broad subject, I encourage you to learn the basics and ask more specific questions as they come up. Your goal is to interpret your filter expression records (fetched from your table) and create a LINQ expression tree for the predicate that they represent. You can then pass the tree to Where() calls as usual.
Without knowing what your UI looks like here is a simple example of what I was talking about in my comments regarding Serialize.Linq library
public void QuerySerializeDeserialize()
{
var exp = "(User.Age > 7 AND User.FirstName == \"Daniel\") OR User.Age < 10";
var user = Expression.Parameter(typeof (User), "User");
var parsExpression =
System.Linq.Dynamic.DynamicExpression.ParseLambda(new[] {user}, null, exp);
//Convert the Expression to JSON
var query = e.ToJson();
//Deserialize JSON back to expression
var serializer = new ExpressionSerializer(new JsonSerializer());
var dExp = serializer.DeserializeText(query);
using (var context = new AppContext())
{
var set = context.Set<User>().Where((Expression<Func<User, bool>>) dExp);
}
}
You can probably get fancier using reflection and invoking your generic LINQ query based on the types coming in from the expression. This way you can avoid casting the expression like I did at the end of the example.

NHibernate Linq Query with Projection and Count error

I have the following query:
var list = repositoy.Query<MyClass>.Select(domain => new MyDto()
{
Id = domain.Id,
StringComma = string.Join(",", domain.MyList.Select(y => y.Name))
});
That works great:
list.ToList();
But if I try to get the Count I got an exception:
list.Count();
Exception
NHibernate.Hql.Ast.ANTLR.QuerySyntaxException
A recognition error occurred. [.Count[MyDto](.Select[MyClass,MyDto](NHibernate.Linq.NhQueryable`1[MyClass], Quote((domain, ) => (new MyDto()domain.Iddomain.Name.Join(p1, .Select[MyListClass,System.String](domain.MyList, (y, ) => (y.Name), ), ))), ), )]
Any idea how to fix that without using ToList ?
The point is, that we should NOT call Count() over projection. So this will work
var query = repositoy.Query<MyClass>;
var list = query.Select(domain => new MyDto()
{
Id = domain.Id,
StringComma = string.Join(",", domain.MyList.Select(y => y.Name))
});
var count = query.Count();
When we use ICriteria query, the proper syntax would be
var criteria = ... // criteria, with WHERE, SELECT, ORDER BY...
// HERE cleaned up, just to contain WHERE clause
var totalCountCriteria = CriteriaTransformer.TransformToRowCount(criteria);
So, for Count - use the most simple query, i.e. containing the same JOINs and WHERE part
If you really don't need the results, but only the count, then you shouldn't even bother writing the .Select() clause. Radim's answer as posted is a good way to both get the results and the count, but if your database supports it, use future queries to execute both in the same roundtrip to the database:
var query = repository.Query<MyClass>;
var list = query.Select(domain => new MyDto()
{
Id = domain.Id,
StringComma = string.Join(",", domain.MyList.Select(y => y.Name))
}).ToFuture();
var countFuture = query.Count().ToFutureValue();
int actualCount = countFuture.Value; //queries are actually executed here
Note that there in NH prior to 3.3.3, this would still execute two round-trips (see https://nhibernate.jira.com/browse/NH-3184), but it would work, and if you ever upgrade NH, you get a (minor) performance boost.

Linq strategy for complex set

Am developing a ViewModel/PresentationModel which is getting complex.
I want the Linq query to return an IQueryable<UserPresentationModel>
Using EntityFramework against MSSQL
Is it possible to do any sort of iteration over the set before returning it to the presentation layer ie
List<UserPresentationModel> list = new List<UserPresentationModel>();
foreach (var person in listOfPeople)
{
UserPresentationModel u = new UserPresentationModel();
int userUIStatus = GetColourStateOfPerson(person);
u.FirstName = person.FirstName;
u.UserUIStatus = userUIStatus;
list.Add(u);
}
return list
This feels like it would always be N+1, and I'd never get the advantages of deferred execution, composing of queries..
Or (and I think am answering my own question) do I need to think in a SQL set based manner.
First, we can convert your code to LINQ.
IEnumerable<UserPresentationModel> models =
from person in listOfPeople
select new UserPresentationModel
{
FirstName = person.FirstName,
UserUIStatus = GetColourStateOfPerson(person)
}
return models.ToList();
Now, if GetColourStateOfPerson is making a DB round-trip, you definitely want to pull that out.
IDictionary<int, int> colourStatesByPersonId = GetColourStatesOfPeople(listOfPeople);
IEnumerable<UserPresentationModel> models =
from person in listOfPeople
select new UserPresentationModel
{
FirstName = person.FirstName,
UserUIStatus = colourStatesByPersonId[person.PersonId]
}
return models.ToList();
You could probably manage to create a single LINQ query that grabs just the first names and colour states of the people you want in a single query, but you haven't provided enough information about your data context for me to help you with that.
I would personally avoid passing around an IQueryable, which could continue making database trips any time somebody touches it. Let your data layer get out all the data you're likely to need, compose it into a list, and return that.
use IEnumerable<T>.Aggregate() instead of looping.
return listOfPeople.Aggregate(new List<UserPresentationModel>(), person => {
return new UserPresentationModel {
FirstName = person.FirstName,
UserUIStatus = GetColourStateOfPerson(person)
};
}).AsQueryable();
return listOfPeople.AsEnumerable().Select(p =>
new UserPresentationModel
{
FirstName = p.FirstName,
UserUIStatus = GetColourStateOfPerson(p)
}).AsQueryable();
I'm assuming that listOfPeople is an IQueryable that will eventually execute against your database. If that is the case then AsEnumerable() is important because SQL Server won't know what to do with GetColourStateOfPerson(). AsEnumerable() will force the IQueryable's expression tree to execute, pull the resulting rows out of your database and then apply Select() transformation in code as oppose to in SQL Server.
If you can implement GetColourStateOfPerson() as a stored proc or database function then you can omit AsEnumerable() and AsQueryable() and allow execution to delay even longer.

NHibernate database call count?

Is there a way to get at runtime the number of calls NHibernate is making to the database?
I know I can use the NHibernate profiler (NHProf, http://nhprof.com/) to manually view this count while debugging - but what I'd like to do is at runtime to get the actual number of calls to the database NHibernate is making so I can throw an exception if it's a ridiculous number like 50 or 300 (exact number to be determined).
This would be an indication to the developer that he/she needs to use 'Eager' and 'Future' and tweak the NHibernate code so hundreds of calls are not being made to the database.
Below I have an example where I'm seeing 284 calls being made to the database -
We can fix this code - so I'm not looking for a solution on how to make this NHibernate code better. I would like instead to implement a system that would notify developers they need to tweak the Nhibernate code.
Example - suppose we have the following model/db -
customer
customeraddress
order
orderstate
orderDetail
The code below makes one select call for each order detail to each of the related tables, etc... With only 72 order details in the db, we end up with 284 calls made to the database.
var query = QueryOver.Of<OrderDetail>()
.Where(c => c.UpdateTime >= updateTime);
using (var context = _coreSessionFactory.OpenSession())
{
var count = query.GetExecutableQueryOver(context)
.ToRowCountQuery()
.FutureValue<Int32>();
var items = query
.OrderBy(a => a.UpdateTime).Desc
.Skip(index ?? 0)
.Take(itemsPerPage ?? 20)
.GetExecutableQueryOver(context)
.Fetch(c => c.Order.OrderState).Eager
.Fetch(c => c.Order.Customer.CustomerAddress).Eager
.Future();
return new
{
Typename = "PagedCollectionOfContract",
Index = index ?? 0,
ItemsPerPage = itemsPerPage ?? 20,
TotalCount = count.Value,
Items = items.ToList().Select(c => new
{
Typename = "OrderDetail",
c.Id,
OrderId = c.Order.Id,
OrderStateAffiliates = c.Order.OrderStates.First(n => n.Name == StateNames.California).AffiliateCount,
CustomerZipCode = c.Order.Customer.CustomerAddresses.First(n => n.StateName == StateNames.California).ZipCode,
CustomerName = c.Order.Customer.Name,
c.DateTimeApproved,
c.Status
})
.ToArray()
};
}
It's not important for this order/customer model to be understood and to improve it - it's just an example so we get the idea on why I need to get the number of calls NHibernate makes to the db.
The SessionFactory can be configured to collect statistics. E.g. the number of successful transactions or the number of sessions that were opened.
This Article at JavaLobby gives some details.
You can use log4net to gather that info.
Logging NHibernate with log4net