I have about 14,000 rows of data. If use the following EF query, its a long time to load because I suspect it is loading all 14,000 rows and only then is any additional filtering done. This is my Select method in my repository.
Public Function SelectAll() As IEnumerable(Of be_Posts) Implements IPostRepository.SelectAll
Dim posts As IEnumerable(Of be_Posts)
Using db As Ctx = New Ctx
posts = db.be_Posts.OrderByDescending(Function(x) x.DateCreated).ToList
Return posts
End Using
And Controller:
Function Index(page As Integer?) As ActionResult
Dim PageSize = System.Web.Configuration.WebConfigurationManager.AppSettings("PageSize")
Dim pageNumber As Integer = If(page, 1)
Dim posts = _repo.SelectAll()
Return View(posts.ToPagedList(pageNumber, PageSize))
End Function
Helper in View:
#Html.PagedListPager((Model), Function(page) Url.Action("Index", New With { _
.page = page _
}), PagedListRenderOptions.ClassicPlusFirstAndLast)
Now If add in a take clause, for example .Take(500) then things are dramatically faster. How can I make this query faster and still work will all the records? I am also using Troy Goode's PagedList extension to get paging functionality. Everything is fine as long as i get just a few hundred records. So what to do? Most if not all examples of paging that I can find that use Troy's library involve all the code directly in the controller.
Calling ToList executes the query and, as you say, it's getting every record. Paging is done using Skip and Take, e.g.
Dim page = list.Skip(pageSize * (pageNumber - 1)).Take(pageSize)
The source list can be the table itself or the result of a Where or OrderBy call or whatever. Just make sure that ToList or, preferably, ToArray is called last.
Related
I fire tasks to download multiple URLs.
Dim downloadTasksQuery As IEnumerable(Of Task(Of Boolean)) =
From company In companies Select DownloadCompanyFromYahooAsync(company, numberOfDays)
' ***Use ToList to execute the query and start the download tasks.
Dim downloadTasks As IEnumerable(Of Task(Of Boolean)) = downloadTasksQuery.ToList()
Await Task.WhenAll(downloadTasks)
The companies list is kind of containing 2000 URLs. I am observing that URLs added towards the end of the list are more frequently timing out. I have retry logics in place and am handling this timeout situation, which downloads the URL on the next try. However, I dont want to give preferential treatment to a URL just because it appears in the beginning of the list.
Hence was trying to think if we can fork 4 main tasks chunking the URL list into 500 each (probably more manageable) and then use the above code. However, am not able to figure out a way to introduce that without having to rewrite too much in the above code. Any help is greatly appreciated.
EDIT:
Something more like this:
Dim chunkPart As OrderablePartitioner(Of Tuple(Of Integer, Integer)) = Partitioner.Create(1, companies.Count, 500)
Parallel.ForEach(chunkPart, Sub(chunkRange)
For i As Integer = chunkRange.Item1 To chunkRange.Item2 - 1
Dim downloadTasksQuery As IEnumerable(Of Task(Of Boolean)) =
From company In companies.Skip(chunkRange.Item1).Take((chunkRange.Item2 - chunkRange.Item1) + 1) Select DownloadCompanyFromYahooAsync(company, numberOfDays)
Dim downloadTasks As IEnumerable(Of Task(Of Boolean)) = downloadTasksQuery.ToList()
Await Task.WhenAll(downloadTasks)
Next
End Sub
This is with minimal code changes, but the issue is that I cannot use Await inside a Parallel.ForEach.
Any suggestions pls to change this.
Not a VB.NET guy, but I think Stephen Toub's good post on implementing a simple ForEachAsync might be helpful to you.
Some code snippet from his post, it allow you to limit the number of operatons that are able to run in parallel.
public static Task ForEachAsync<T>(this IEnumerable<T> source, int dop, Func<T, Task> body)
{
return Task.WhenAll(
from partition in Partitioner.Create(source).GetPartitions(dop)
select Task.Run(async delegate {
using (partition)
while (partition.MoveNext())
await body(partition.Current);
}));
}
For your specific question, you can then use this as such:
public async Task DownloadForAllCompanies(List<string> companies, int numberOfDays)
{
await companies.ForEachAsync(4, async company =>
{
await DownloadCompanyFromYahooAsync(company, numberOfDays);
});
}
I have found this example on StackOverflow:
var people = new List<Person> {
new Person{Name="aaa", Salary=15000, isHip=false}
,new Person{Name="aaa", Salary=15000, isHip=false}
,new Person{Name="bbb", Salary=20000, isHip=false}
,new Person{Name="ccc", Salary=25000, isHip=false}
,new Person{Name="ddd", Salary=30000, isHip=false}
,new Person{Name="eee", Salary=35000, isHip=false}
};
people.Where(p => p.Salary < 25000).Update(p => p.isHip = true);
foreach (var p in people)
{
Console.WriteLine("{0} - {1}", p.Name, p.isHip);
}
public static void Update<T>(this IEnumerable<T> source, Action<T> action)
{
foreach (var item in source)
action(item);
}
In C# everything works fine.
I tried to convert it in VB.NET.
Here's the code:
<System.Runtime.CompilerServices.Extension()> _
Public Sub Update(Of T)(ByVal source As IEnumerable(Of T), ByVal action As Action(Of T))
For Each item In source
action(item)
Next item
End Sub
If I try to update my collection things don't work, though:
people.Where(Function(p) p.Salary < 25000).Update(Function(p) p.isHip = true)
I am using VS2008 (3.5)
This thing is driving me crazy.
Is there anybody who can help me?
Alberto
You should always post what exactly is not working.
In your case, you want to Update list elements, which works though passing an Action(Of T) that should be run for every element.
Such an action, that is just run, performs some side-effects but returns no value is described by exactly one VB construct: A Sub.
Thus what you would want to write is
.Update(Sub(p) p.isHip = true)
which is valid VB2010, but simply does not work in the 2008 version. C# doesn't have a problem there, but in your VB code, you want to pass a Function which has to produce a value and not just perform an assignment. Func(Of ...) would be the appropriate type of that expression.
So what to do?
You can't just express what you want in the syntax of your version. But probably you shouldn't - build a new collection without modifying an old one. Is soon as you're dealing with value types/properties, the above approch won't work at all, since actually a temporary collection returned by Where is modified. Linq is no modification language, but a query system.
Anyway: Just use a plain loop.
I am trying to load a sortable jqgrid 3.5 from a query with multiple joins in it and having much difficulty as I am a novice with both Linq and jqgrid. In order to allow for sorting I first was attempting to load it using dynamic SQL.
Since I am pulling columns from multiple tables I assume my return will be a class object which I will populate (or will it be a table). How can I return a IQueryable custom class object when using dynamic SQL with multiple .JOIN clauses. If this is impossible how do I return IQueryable data from a stored procedure call. It is easy to create dynamic SQL in the stored procedure - I am unsure how to load my grid with it however.
Sorry if this is all over the place but I can't seem to find a way. If you can recommend the most straight forward way to load my sortable grid from a query which has multiple joins in I am much appreciated.
My controller code:
public ActionResult GridData(string sidx, string sord, int page, int rows)
{
EquipTrak eqt = new EquipTrak();
var equipment = eqt.GetGridEquipment(sidx, sord);
var dataJson = new
{
total = 10000,
page = 1,
records = 10000,
rows = (from e in equipment
select new
{
equip_id = e.equip_id,
cell = new string[] {
e.equip_id,
e.equipType,
e.makeType,
String.Format("{0:MM/dd/yyyy}", e.serv_due_dt)
}
}).ToArray()
};
return Json(dataJson);
}
}
my class code (incomplete):
namespace ULS_Site.Models
{
public class EquipTrak
{
uls_dbDataContext ulsDB = new uls_dbDataContext();
public IQueryable<equipmentCls> GetGridEquipment(string sidx, string sord)
{
try
{
return
Not sure if this is the best or worst solution but I used SQL Server views to handle all the joining required. I could then use .Orderby and .Where against the view which was in my data context.
I was wondering if there is any way to intercept and modify the sql generated from linq to Sql before the query is sent off?
Basically, we have a record security layer, that given a query like 'select * from records' it will modify the query to be something like 'select * from records WHERE [somesecurityfilter]'
I am trying to find the best way to intercept and modify the sql before its executed by the linq to sql provider.
Ok, first to directly answer your question (but read on for words of caution ;)), there is a way, albeit a finicky one, to do what you want.
// IQueryable<Customer> L2S query definition, db is DataContext (AdventureWorks)
var cs = from c in db.Customers
select c;
// extract command and append your stuff
DbCommand dbc = db.GetCommand(cs);
dbc.CommandText += " WHERE MiddleName = 'M.'";
// modify command and execute letting data context map it to IEnumerable<T>
var result = db.ExecuteQuery<Customer>(dbc.CommandText, new object[] { });
Now, the caveats.
You have to know which query is generated so you would know how to modify it, this prolongs development.
It falls out of L2S framework and thus creates a possible gaping hole for sustainable development, if anyone modifies a Linq it will hurt.
If your Linq causes parameters (has a where or other extension causing a WHERE section to appear with constants) it complicates things, you'll have to extract and pass those parameters to ExecuteQuery
All in all, possible but very troublesome. That being said you should consider using .Where() extension as Yaakov suggested. If you want to centrally controll security on object level using this approach you can create an extension to handle it for you
static class MySecurityExtensions
{
public static IQueryable<Customer> ApplySecurity(this IQueryable<Customer> source)
{
return source.Where(x => x.MiddleName == "M.");
}
}
//...
// now apply it to any Customer query
var cs = (from c in db.Customers select c).ApplySecurity();
so if you modify ApplySecurity it will automatically be applied to all linq queries on Customer object.
If you want to intercept the SQL generated by L2S and fiddle with that, your best option is to create a wrapper classes for SqlConnection, SqlCommand, DbProviderFactory etc. Give a wrapped instance of SqlConnection to the L2S datacontext constructor overload that takes a db connection. In the wrapped connection you can replace the DbProviderFactory with your own custom DbProviderFactory-derived class that returns wrapped versions of SqlCommand etc.
E.g.:
//sample wrapped SqlConnection:
public class MySqlConnectionWrapper : SqlConnection
{
private SqlConnecction _sqlConn = null;
public MySqlConnectionWrapper(string connectString)
{
_sqlConn = new SqlConnection(connectString);
}
public override void Open()
{
_sqlConn.Open();
}
//TODO: override everything else and pass on to _sqlConn...
protected override DbProviderFactory DbProviderFactory
{
//todo: return wrapped provider factory...
}
}
When using:
using (SomeDataContext dc = new SomeDataContext(new MySqlConnectionWrapper("connect strng"))
{
var q = from x in dc.SomeTable select x;
//...etc...
}
That said, do you really want to go down that road? You'll need to be able to parse the SQL statements and queries generated by L2S in order to modify them properly. If you can instead modify the linq queries to append whatever you want to add to them, that is probably a better alternative.
Remember that Linq queries are composable, so you can add 'extras' in a separate method if you have something that you want to add to many queries.
first thing come to my mind is to modify the query and return the result in Non-LINQ format
//Get linq-query as datatable-schema
public DataTable ToDataTable(System.Data.Linq.DataContext ctx, object query)
{
if (query == null)
{
throw new ArgumentNullException("query");
}
IDbCommand cmd = ctx.GetCommand((IQueryable)query);
System.Data.SqlClient.SqlDataAdapter adapter = new System.Data.SqlClient.SqlDataAdapter();
adapter.SelectCommand = (System.Data.SqlClient.SqlCommand)cmd;
DataTable dt = new DataTable("sd");
try
{
cmd.Connection.Open();
adapter.FillSchema(dt, SchemaType.Source);
adapter.Fill(dt);
}
finally
{
cmd.Connection.Close();
}
return dt;
}
try to add your condition to the selectCommand and see if it helps.
Try setting up a view in the DB that applies the security filter to the records as needed, and then when retrieving records through L2S. This will ensure that the records that you need will not be returned.
Alternatively, add a .Where() to the query before it is submitted that will apply the security filter. This will allow you to apply the filter programmatically (in case it needs to change based on the scenario).
I have been writing same code for insert, update, delete with LINQ over and over again. I want to have some sort of generic function for Insert, Update, Delete operation. I read a post here like the following :
public static void Insert<T>(T entity) where T : class
{
using (OrcasDB database = new OrcasDB())
{
database.GetTable<T>().Add(entity);
database.SubmitChanges();
}
}
public static void Delete<T>(Expression<Func<T, bool>> predicate)
where T : class
{
using (OrcasDB database = new OrcasDB())
{
T instance = (T) database.GetTable<T>().Where<T>(predicate).Single();
database.GetTable<T>().Remove(instance);
database.SubmitChanges();
}
}
How to Use
// insert
Employee will = new Employee
{
Username = "will.asrari",
EmailAddress = "me#willasrari.com",
CanCode = true
};
LinqHelper.Insert<Employee>(will);
// delete
LinqHelper.Delete(emp => emp.EmployeeId.Equals(3));
Yes, I would like to write something like in VB.NET. Is the code above good to follow? Can anyone show me any LINQ to SQL generic class for Insert, Delete, Update written in VB.NET?
Thank you.
FYI, I managed to write a simple class to do the generic CUD operantion for LINQ to SQL.
'Class GenericCUD.vb
Imports System.Linq.Expressions
Imports System.Data.Linq
Public Class GenericCUD
Public Shared Sub Insert(Of T As Class)(ByVal theEntity As T)
Using db As New DemoDataContext()
db.GetTable(Of T)().InsertOnSubmit(theEntity)
db.SubmitChanges()
End Using
End Sub
Public Shared Sub Update(Of T As Class)(ByVal originalEntity As T, ByVal newEntity As T)
Using db As New DemoDataContext()
db.GetTable(Of T)().Attach(newEntity, originalEntity)
db.Refresh(RefreshMode.KeepCurrentValues, newEntity)
db.SubmitChanges()
End Using
End Sub
Public Shared Sub Delete(Of T As Class)(ByVal theEntity As T)
Using db As New DemoDataContext()
db.GetTable(Of T)().Attach(theEntity)
db.GetTable(Of T).DeleteOnSubmit(theEntity)
db.Refresh(RefreshMode.KeepCurrentValues, theEntity)
db.SubmitChanges()
End Using
End Sub
End Class
How to use the class :
'Using Insert
Dim ta As New TestAuthor
ta.FirstName = TextBox1.Text
ta.LastName = TextBox2.Text
GenericCUD.Insert(ta)
'Using Update
Dim original As New TestAuthor
original.Id = 3
Dim newEntity As New TestAuthor
newEntity.Id = original.Id
newEntity.FirstName = TextBox1.Text
newEntity.LastName = TextBox2.Text
GenericCUD.Update(original, newEntity)
'Using Delete
Dim ta As New TestAuthor
ta.Id = 7
GenericCUD.Delete(ta)
I read a lot of post on many blogs. Here are a few that really helped me to make the GenericCUD work:
LINQ, Lambda, and Generics: Insert and Delete
LINQ to SQL CRUD
How to Make LINQ to SQL Check for Changes After Attach
So, What do you think about the GernericCUD class above? Please give me some comment because I want to improve it. Thank you.
We've taken a similar approach in our 3-tier application framework. We currently have roughly 80 entities and have used generics to create a very light-weight set of generic CRUD methods that satifsy those 80 entities and any number of entities.
The only suggestion I might make is to re-think your approach to creating a new database context for each insert, update and delete operation. The problem is that if you need to wrap multiple inserts, updates and/or deletes in a single transaction, you're going to need to use a TransactionScope object because each insert/update/delete is using it's own context object. Using TransactionScope is ok, but since you've got multiple connections, the transaction is going to get elevated to an MTC transaction, which is a hassle.
Can't help you with the VB code. IMO, learn and stick with C#.
Randy