We're having alot of troubles here with the .ToList command, it's used in VB.NET with a MVC ASP.NET web project.
We have ~2000 entries in our database, we use a LINQ command to SELECT and ORDER the 2000 entries. The result is transformed into a list by the .ToList method for our pager and grid builder. Problem is, the .ToList takes WAY WAY TOO long (we're talking 40-60seconds to execute) so our websites looks slow as hell.
We tested the equivalent SQL command on the database and it responds quickly. It's not a problem with the commands or a slow database server. We tried an IEnumrable witch was alot faster but we need it in the .ToList format at the end for our grids. What's the deal with the .ToList ? Anything we can do ?
Here's the code :
'list = (From c In _entities.XXXXXXXXSet.Include("XXXXXX").Include("XXXXXX") _
Where Not (c.XXXXXX Is Nothing AndAlso c.XXXXXX = String.Empty) _
And c.XXXXXX = codeClient _
And c.XXXXXX > dateLimite _
Order By c.XXXXXX Descending _
Select c).ToList()
We divided the code and to leave only the .ToList function alone and that's really what sucks up all the time. The LINQ command executes in no time.
Thanks alot.
Tom
Of course the LINQ command "executes" in no time, because it just represents the query. The query is only executed once you iterate over it, which is exactly what the ToList method does.
I would advise you to use the Skip and Take operators in your pagers to narrow down the result queried from the database. Doing this, you only request the 10 or 20 elements or whatever you need, resulting in a much smoother experience.
I think it would be better to page in the query instead of fetching all data in one go, using Skip and Take.
list = (From c In _entities.XXXXXXXXSet.Include("XXXXXX").Include("XXXXXX") _
Where Not (c.XXXXXX Is Nothing AndAlso c.XXXXXX = String.Empty) _
And c.XXXXXX = codeClient _
And c.XXXXXX > dateLimite _
Order By c.XXXXXX Descending _
Select c).Skip(pageSize * pageIndex).Take(pageSize).ToList();
That, paired with some well targeted caching (if possible) should provide a snappier user experience.
When you say "the equivalent SQL command on the database and it responds quickly" - is that the actual SQL statements which the LINQ code is generating or handcoded SQL which is logically equivalent?
Because that LINQ-generated code might not be terribly efficient.
For stuff like this, it's often useful to run the code in the profiler. There could be any number of things slowing down... network, memory, object size, etc.
You could also create your own list and copy the IEnumerable values into it. If it's at all possible, I would recommend changing your grid to accept an IEnumerable.
To confirm the performance of ToList as opposed to query execution, add a statement and compare:
//this call iterates a query, causing a database roundtrip.
List<Row> result = query.ToList();
//this call generates a new List by iterating the old List.
result = result.ToList();
Looking over your query, I suspect you'll need an codeClient, and an index on each of the tables mentioned in the calls to .Include. Grab the generated sql and check the execution plan to confirm.
Related
Context
I have a interface in VB.NET that extract the data from the UniVerse using UniObjects for .NET
Problem
From the COB file I need to get all keys where the FEC.COB field is equal to a specific date and the field SEC is equal to 04.
An expert in UniVerse Database told me that I can run the follow queries:
SELECT COB WITH FEC.COB > “31/10/2013”
SELECT.ID 1 2 04
But I don't know how can I do that with UniObjects library. Can anyone help me?
I don't use UniObjects as my shop normally gets data our of UniVerse via ODBC. Also my VB is bad, so I don't have much metacode for you, but the basic idea would be to do something like this.
1.) Create a UV Session. Hopefully you have that much worked out as I can be of next to no help there.
2.) Once the session is established Execute your query by doing something like this
session.Command.Text = "SELECT COB WITH FEC.COB > '31/10/2013'"
session.Command.Exec
(I converted your double quotes to single quotes and Universe won't mind).
3.) If you just need the IDs, you can get them by iterating through the select list that your query returns. A command line query will always return to list 0 unless you specify otherwise in your UV query. In most cases your results will be in session.SelectList(0)
Dim objSelect As object
Set objSelect = objSession.SelectList(0)
4.) It looks like the SelectList object has a ReadList method which returns a Dynamic Array Object, which you should be able to iterate through using normal array looping. Additionally you can use a while loop and next to do what you need to do.
Dim someObject as Object
someObject = objSelect.Next ' Get first ID
Do While Not objSelect.LastRecordRead
' Do something here with someObject. Maybe ToString it or something
someObject = objSelect.Next' Get next ID
Loop
Hope that is somewhat helpful.
Following a tutorial, i've built a LINQ query to filter a List of objects:
Dim Query = From FilteredItem In Items, FiltSupp In FilteredItem.Suppliers
Where ((SuppliersComboBox.SelectedItem.ToString = "All Suppliers" OrElse FiltSupp.CompanyName = SelectedSupplier) And
(String.IsNullOrEmpty(TxtItemCode.Text) OrElse FilteredItem.ItemCode.ToString.Contains(TxtItemCode.Text)) And
(String.IsNullOrEmpty(TxtName.Text) OrElse FilteredItem.Name.ToLower.Contains(TxtName.Text.ToLower)) And
(String.IsNullOrEmpty(TxtPkgCost.Text) OrElse FilteredItem.PkgCost.ToString.Contains(TxtPkgCost.Text)))
Select FilteredItem
The problem is that the query returns nothing even when the list is being populated.
At debugging, the list value is correct (populated with few items), but the query doesn't returning anything. I can't figure out what is wrong, so any suggestion would be greatly appreciated!
Ps. I'm using vb.net, but answers in C# are also wellcome
To start off with, this answer is obviously not complete yet, so please hold off any downvotes until I finalize it. I have every intention of helping the OP if I can, however at this time there is no way for me to even begin helping out because the information needed is not present.
#ISAE, Can you please provide the following:
hoping the Items being queried is a strongly typed DataTable; the class file for the Items object type, preferably the designer files also (.xsc, .xsd, and .xss)
the forms' designer files and code behinds.
Alternatively, if the project can be put on GitHub, that would be even better. If you can't release all of that code, then really need to know enough to somewhat reconstruct what's going on. As one of the comments pointed out, that LINQ query is really messy and it's hard to dissect right now.
If Not m_Batchs Is Nothing Then
For Each Batch In m_Batchs
newListItem = lstWsJobs.Items.Add(Batch.Id.ToString)
With newListItem
.Name = Batch.Id.ToString()
.SubItems.Add(Batch.JobId.ToString)
.SubItems.Add(Batch.Complete.ToString)
.SubItems.Add(Batch.User)
.SubItems.Add(Batch.Time.ToString)
End With
Next
End If
I have this list view (which is working fine) and i want to find an efficient way of populating it in a specific order, ie by date, by identity etc.
I know i can use linq but as i understand this is inefficient. If m_batchs is a large list of objects then i will looping through this list many, many time (as linq behind the scenes loops through the object collection).
Any ideas?
LINQ is not inefficient in general but almost always it's easier to read and faster to implement, change and extend. Also, does it really matter if one approach is 1 millisecond faster on 1000 iterations?
So i assume that Batch is a custom type and m_Batchs is a List<Batch>:
// order by date
var query = m_Batchs.OrderBy(b => b.Time);
// order by identity
query = m_Batchs.OrderBy(b => b.ID);
// ...
Measure the difference between this simple LINQ query and your custom implementation.
Edit: Sorry, that was C#
Dim batchByTime = m_Batchs.OrderBy(Function(b) b.Time);
Dim batchByID = m_Batchs.OrderBy(Function(b) b.ID);
Given a datatable containing two columns like this:
Private Function CreateDataTable() As DataTable
Dim customerTable As New DataTable("Customers")
customerTable.Columns.Add(New DataColumn("Id", GetType(System.Int32)))
customerTable.Columns.Add(New DataColumn("Name", GetType(System.String)))
Dim row1 = customerTable.NewRow()
row1.Item("Id") = 1
row1.Item("Name") = "Customer 1"
customerTable.Rows.Add(row1)
Dim row2 = customerTable.NewRow()
row2.Item("Id") = 2
row2.Item("Name") = "Customer 2"
customerTable.Rows.Add(row2)
Dim row3 = customerTable.NewRow()
row3.Item("Id") = 3
row3.Item("Name") = "Customer 3"
customerTable.Rows.Add(row3)
Return customerTable
End Function
Would you use this snippet to retrieve a List(Of Integer) containing all Id's:
Dim table = CreateDataTable()
Dim list1 As New List(Of Integer)
For i As Integer = 0 To table.Rows.Count - 1
list1.Add(CType(table.Rows(i)("Id"), Integer))
Next
Or rather this one:
Dim list2 = (From r In table.AsEnumerable _
Select r.Field(Of Integer)("Id")).ToList()
This is not a question about whether to type cast the Id column to Integer by using .Field(Of Integer), CType, CInt, DirectCast or whatever but generally about whether or not you choose Linq over forloops as the subject implies.
For those who are interested: I ran some iterations with both versions which resulted in the following performance graph:
graph http://dnlmpq.blu.livefilestore.com/y1pOeqhqQ5neNRMs8YpLRlb_l8IS_sQYswJkg17q8i1K3SjTjgsE4O97Re_idshf2BxhpGdgHTD2aWNKjyVKWrQmB0J1FffQoWh/analysis.png?psid=1
The vertical axis shows the milliseconds it took the code to convert the rows' ids into a generic list with the number of rows shown on the horizontal axis. The blue line resulted from the imperative approach (forloop), the red line from the declarative code (linq).
Whatever way you generally choose: Why do you go that way and not the other?
Whenever possible I favor the declarative way of programming instead of imperative. When you use a declarative approach the CLR can optimize the code based on the characteristics of the machine. For example if it has multiple cores it could parallelize the execution while if you use an imperative for loop you are basically locking this possibility. Today maybe there's no big difference but I think that in the future more and more extensions like PLINQ will appear allowing better optimization.
I avoid linq unless it helps readability a lot, because it completely destroys edit-and-continue.
When they fix that, I will probably start using it more, because I do like the syntax a lot for some things.
For almost everything I've done I've come to the conclusion that LINQ is optimized enough. If I handcrafted a for loop it would have better performance, but in the grand scheme of things we are usually talking milliseconds. Since I rarely have a situation where those milliseconds will make any kind of impact, I find it's much more important to have readable code with clear intentions. I would much rather have a call that is 50ms slower than have someone come along and break it altogether!
Resharper has a cool feature that will flag and convert loops into Linq expressions. I will flip it to the Linq version and see if that hurts or helps readability. If the Linq expression more clearly communicates the intent of the code, I will go with that. If the Linq expression is unreadable, I will flip back to the foreach version.
Most of the performance issues don't really compare with readability for me.
Clarity trumps cleverness.
In the above example, I would go with the the Linq version since it clearly explains the intent and also locks out people accidently adding side effects in the loop.
I recently found myself wondering whether I've been totally spoiled by LINQ. Yes, I now use it all the time to pick all sort of things out from all sort of collections.
I started to, but found out in some cases, I saved time by using this approach:
for (var i = 0, len = list.Count; i < len; i++) { .. }
Not necessarily in all cases, but some. Most extension methods use the foreach approach of querying.
I try to follow these rules:
Whenever I'm just querying (filtering, projecting, ...) collections, use LINQ.
As soon as I'm actually 'doing' something with the result (i.e, introduce side effects), I'll use a for loop.
So in this example, I'll use LINQ.
Also, I always try to split up the 'query definition' from the 'query evaluation':
Dim query = From r In table.AsEnumerable()
Select r.Field(Of Integer)("Id")
Dim result = query.ToList()
This makes it clear when that (in this case in-memory) query will be evaluated.
I'm working on my first project using LINQ (in mvc), so there is probably something very simple that I missed. However, a day of searching and experimenting has not turned up anything that works, hence the post.
I'm trying to write a LINQ query (Linq to SQL) that will contain a multiple number of conditions in the where statement separated by an OR or an AND. We don't know how many conditions are going to be in the query until runtime. This is for a search filter control, where the user can select multiple criteria to filter by.
select * from table
where table.col = 1
OR table.col = 2
OR table.col = 7
.... 'number of other conditions
Before I would just construct the SQL query as a string while looping over all conditions. However, it seems like there should be a nice way of doing this in LINQ.
I have tried looking using expression trees, but they seem a bit over my head for the moment. Another idea was to execute a lambda function inside the where statement, like so:
For Each value In values
matchingRows = matchingRows.Where(Function(row) row.col = value)
However, this only works for AND conditions. How do I do ORs?
I would use PredicateBuilder for this. It makes dynamic WHERE clauses very easy.
AND is easy - you can just call Where in a loop. OR is much trickier. You mention SQL, so I'm assuming this is something like LINQ-to-SQL, in which case one way I've found to do this involves building custom Expression trees at runtime - like so (the example is C#, but let me know if you need help translating it to VB; my VB isn't fantastic any more, so I'll let you try first... you can probably read C# better than I can write VB).
Unfortunately, this won't work with EF in 3.5SP1 (due to the Expression.Invoke), but I believe this is fixed in 4.0.
Something like this should work (forgive my VB):
Expression(Of Func(Of Something, Boolean)) filter = Nothing
ParameterExpression rowParam = Expression.Parameter("row", CType(Something))
For Each value In values
filterPart = Expression.Equal( _
Expression.Property(rowParam, "col"), _
Expression.Constant(value)))
If filter Is Nothing Then
filter = filterPart
Else
filter = Expression.OrElse(filter, filterPart)
End If
Next
If newPredicate IsNot Nothing Then
matchingRows = matchingRows.Where( _
Expression.Lambda(Of Func(Of SomeType, Boolean))(filter, rowParam))
End If
No guarantees, however, my VB is a little rusty :-)
But PredicateBuilder might be a better solution if you want to do more complicated stuff than just Ands and Ors.