Time out retrieving items from list - sql

Question

I have a problem retrieving items from a fairly large list. I can quickly and easily retrieve items from a small list with more or less 50 items, but when I try to retrieve items from a list containing more or less 4600 items, the sqlsever.exe process spikes for the duration of the request, but the items are never retrieved. If have set up the web applications throttling settings, so it can’t be that which is causing the problem. Here is the code that I originally used to retrieve the items. There is really nothing special to it.
using (SPSite site = new SPSite(siteUrl))
{
using (SPWeb web = site.OpenWeb())
{
SPList list = web.Lists[uid.ToString()];
SPListItemCollection itemCollection = list.Items;
foreach (SPListItem i in itemCollection) //This is where the code stops responding
{
//Use list items
}
}
}
After that didn’t work, I tried a couple of other methods to retrieve the items from the list. Here is the code:
SPList list = web.Lists[uid.ToString()];
SPQuery query = new SPQuery();
query.Query = "";
query.QueryThrottleMode = SPQueryThrottleOption.Override;
SPListItemCollection itemCollection = list.GetItems(query);
//The code stops here
//I added this part for interest sake, i wanted to if it was the looping that caused the problem
//It seems the when you try to access properties of the item collection that the problem occurs
int itemCount = itemCollection.Count;
foreach (SPListItem i in itemCollection)
{
//Use list items
}
I also tried:
SPList list = web.Lists[uid.ToString()];
SPListItemCollectionPosition pos;
DataTable dt = list.GetDataTable(new SPQuery(), SPListGetDataTableOptions.None, out pos); //The code stops responding here
foreach (DataRow i in dt.Rows)
{
//Use data rows
}
Does anyone know what could be causing this problem?
Thank you in advance!

Answer 1

After a very long struggle we found a solution.
We found this post:
http://trycatch.be/blogs/tom/archive/2009/04/22/never-turn-off-quot-auto-create-amp-auto-update-statistics-quot.aspx
We tested it and it worked!!!
So all we had to do was switch "Auto create statistics" and "Auto update statistics" to true, and to problem was solved
Thanks for all the replys

Answer 2

You'll kill your server if you always try to retrieve all items either through list.Items, list.GetItems(query) (with an empty query).
You need to define a relevant query and specify the amount of results you want to retrieve through the RowLimit property of your SPQuery
SPQuery query = new SPQuery();
query.Query = "<OrderBy><FieldRef Name='Title' /></OrderBy>"; // any relevant query here
query.RowLimit = 50;
Failing to do so will load all the items in memory prior to any other operation. You will probably kill your application pool available memory or your sql server memory when it will try to load all these data !
Hope that helped.

Answer 3

Try to get data using CAML query.
Once you have data you can put that in SPQuery.

Answer 4

Try to load the items not all at the same time. You can load them bulk wise (page wise) with the help of SPQuery.ListItemCollectionPosition (http://msdn.microsoft.com/en-us/library/microsoft.sharepoint.spquery.listitemcollectionposition.aspx).