Getting 2100 parameter limit in LINQ - vb.net

I am getting 2100 parameter limit error in a LINQ to SQL query. Here is the procedure which returns the list of id's. Parameter l has over 5000 id's.
Public Function GetByID(ByVal l As List(Of Int32)) As List(Of OfficeAccess)
Return (From d In db.OfficeAccess
Where l.Contains(d.ID)
Order By d.ID Ascending Select d).ToList()
End Function
I have tried the solution given in the second part of Hitting the 2100 parameter limit (SQL Server) when using Contains() but I'd need another solution without manually building SQL query string and this doesn't work:
Dim ids = String.Join(" ", l.ToArray())
Return(From d In db.OfficeAccess
Where ids.IndexOf(Convert.ToString(d.ID)) != -1
Order By d.ID Ascending Select d).ToList()
Updated
I have used the following code and works fine.
Dim l As New List(Of OfficeAccess)
Dim f As Int32, t As List(Of Int32)
While (ids.Count > 0)
If (ids.Count < 2000) Then f = ids.Count Else f = 2000
t = ids.GetRange(0, f)
l.AddRange((From d In db.OfficeAccess Where t.Contains(d.ID) Select d).ToList())
ids.RemoveRange(0, f)
End While

You may want to partition your original list into smaller chunks. Let's say each one of 2000 elements, this will affect performance (especially if SQL Server version doesn't support OFFSET n and FETCH NEXT m ROWS) but it'll solve your problem.
Function Partition(ByVal l As List(Of Integer)) As List(Of OfficeAccess)
Const size As Integer = 2000
Dim result As List(Of OfficeAccess) = new List(Of OfficeAccess)
Dim index As Integer = 1
Dim partition As List(Of Integer) = (
From item In l
Take size
).ToList()
While partition.Any()
result.AddRange(
From d In db.OfficeAccess
Where partition.Contains(d.ID)
Order By d.ID Ascending
Select d
)
partition = (
From item in l
Skip index * size
Take size
).ToList()
index += 1
End While
Return result
End Function
Please note code is untested then be ready to syntax errors!
Note: in this example I accumulate partial results into result list but you may concatenate each enumeration with Enumerable.Concat() (declaring result as IEnumerable(Of OfficeAccess) and performing ToList() once: Return result.ToList(). It shouldn't make any (visible) difference unless you have a huge number of records (because, at least, you avoid to resize result for each bulk insertion) or unless LINQ is smart enough to optimize (somehow) full resulting query (as concatenation of multiple queries).
I'd avoid, if possible, to manually build SQL string (also to keep code reusable for other data types and easy to update and refactor) but, from linked post, you should try code from one of answers instead of not working code from question...

Related

How to return a List(Of String) from a LINQ statement with a Group By in VB.net?

I've seen several questions on how to do this in C# but I'm having trouble translating those to VB. Here's the basics of my issue:
Table of data NOT normalized and accessed via Entity Framework
Get all unique string values in a certain field
Convert those values to a List(Of String)
This works but I'm guessing there's a better way to do it without iterating through the list:
Public Function GetGroups() As IEnumerable(Of String)
Dim GroupList As New List(Of String)
Dim CodeList = (From c In Context.Codes
Group c.Group By c.Group Into g = Group)
For Each c In CodeList
GroupList.Add(c.Group)
Next
Return GroupList
End Function
What I seem to be struggling with the most is using Group By in LINQ. I'm guessing this could probably be done in 1 or 2 lines by having LINQ return just the list of strings or by converting the list of anonymous objects to a list of strings.
Well, if you don't need anything in the group, you can just use .Distinct():
Return (
From c In Context.Codes
Order By c.Group
Select c.Group
).Distinct().ToList()
Edit: Added Order By

Distinct ComboBox Items

I am trying to convert sql to entity and I need to select distinct items. I thought this would work but its returning all the rows instead of the distinct items.
Dim OrderNos = (From r In Orders.R3Delivery Where r.mainOrderNumber <> "" Select r).Distinct().ToList()
For Each thisentry In OrderNos
cbOrderNumbers.DisplayMember = thisentry.mainOrderNumber
cbOrderNumbers.ValueMember = thisentry.mainOrderNumber
Next
Also is their any good free sql to linq tools out their linquer good but its like 60 quid
The problem is that the Distinct() is comparing the entire object being returned, not just the order number.
If you only need the order numbers, changing this line should get you there:
Dim OrderNos = (From r
In Orders.R3Delivery
Where r.mainOrderNumber <> ""
Select r.mainOrderNumber).Distinct().ToList()
If you need the whole object, then it gets more complicated.

Error when using linq on datatable

I am trying to run the following code converting my datatable to be usable in linq all seems fines and compiles but when I Execute the statement I get the following statement i get the error below new entires just has location and ordernumber in the return values I have to do it this way as I am supporting a legacy access 97 system thanks.
Dim total = From row In newEntries.AsEnumerable()
Select row.Field(Of Int32)("location") Distinct
retVal = Convert.ToInt32(total)
This is my whole code but im still getting an invalid type cast error their is data exsits for this order by teh way
Dim retVal As Int32
Dim newEntries As New DataTable
Dim script As String = scriptBuilder.GetDistinctOrdersForLocations(OrderNumber)
newEntries = connection.SqlSelectToDataTable(script)
Dim total = From row In newEntries.AsEnumerable()
Select row.Field(Of Int32)("location") Distinct
retVal = total.Count()
If you want the count of the collection just do this:
retVal = total.Count()
this will return the count from the distinct query that you have written.
Just to clarify, #David B identified the data type of location was int16 not int32, so changing this in the linq query resolved the issue.
Your LINQ query is returning a collection. You should use something like First or FirstOrDefault.
I'm a little rusty on VB LINQ, but try :
retVal = Convert.ToInt32(total.First())
Note: This will throw an error if there are no items in the collection.
It's important to understand when you write a LINQ query and assign it to a variable, that variable essentially contains a query object, and not the results of running the query. In order to get the value that results from the query, you need to call some method on the query object such as:
total.Single() ' Assumes you expect the query to return exactly one result.
I changed the code to int16 worked here is the code for any one else stuck thanks #Ric
Dim retVal As Int32
Dim newEntries As New DataTable
Dim script As String = scriptBuilder.GetDistinctOrdersForLocations(OrderNumber)
newEntries = connection.SqlSelectToDataTable(script)
Dim total = From row In newEntries.AsEnumerable()
Select row.Field(Of Int16)("location") Distinct
retVal = total.Count()

VB.Net - Efficient way of de-duplicating data

I am dealing with a legacy application which is written in VB.Net 2.0 against a SQL 2000 database.
There is a single table which has ~125,000 rows and 2 pairs of fields with similar data.
i.e. FieldA1, FieldB1, FieldA2, FieldB2
I need to process a combined, distinct list of FieldA, FieldB.
Using SQL I have confirmed that there are ~140,000 distinct rows.
Due to a very restrictive framework in the application I can only retrieve the data as either 2 XML objects, 2 DataTable objects or 2 DataTableReader objects. I am unable to execute custom SQL using the framework.
Due to a very restrictive DB access policy I am unable to add a View or Stored Proc to retrieve as a single list.
What is the most efficient way to combine the 2 XML / DataTable / DataTableReader objects into a single, distinct, IEnumerable object for later processing?
I may have missed something here but could you not combine both DataTables using Merge?
DataTableA.Merge(DataTableB)
You can then use DataTableA.AsEnumerable()
Then see this answer on how to remove duplicates or
You can do this with a DataView as follows: dt.DefaultView.ToTable(True,[Column names])
This is the solution I came up with.
Combine the 2 DataTables using .Merge (thanks to Matt's answer)
Using this as a base I came up with the following code to get distinct rows from the DataTable based on 2 columns:
Private Shared Function GetDistinctRows(sourceTable As DataTable, ParamArray columnNames As String()) As DataTable
Dim dt As New DataTable
Dim sort = String.Empty
For Each columnName As String In columnNames
dt.Columns.Add(columnName, sourceTable.Columns(columnName).DataType)
If sort.Length > 0 Then
sort = sort & ","
End If
sort = sort & columnName
Next
Dim lastValue As DataRow = Nothing
For Each dr As DataRow In sourceTable.Select("", sort)
Dim add As Boolean = False
If IsNothing(lastValue) Then
add = True
Else
For Each columnName As String In columnNames
If Not (lastValue(columnName).Equals(dr(columnName))) Then
add = True
Exit For
End If
Next
End If
If add Then
lastValue = dr
dt.ImportRow(dr)
End If
Next
Return dt
End Function

How do I implement pagination in SQL for MS Access?

I'm accessing a Microsoft Access 2002 database (MDB) using ASP.NET through the OdbcConnection class, which works quite well albeit very slowly.
My question is about how to implement pagination in SQL for queries to this database, as I know I can implement the TOP clause as:
SELECT TOP 15 *
FROM table
but I am unable to find a way to limit this to an offset as can be done with SQL Server using ROWNUMBER. My best attempt was:
SELECT ClientCode,
(SELECT COUNT(c2.ClientCode)
FROM tblClient AS c2
WHERE c2.ClientCode <= c1.ClientCode)
AS rownumber
FROM tblClient AS c1
WHERE rownumber BETWEEN 0 AND 15
which fails with:
Error Source: Microsoft JET Database Engine
Error Message: No value given for one or more required parameters.
I can't work out this error, but I'm assuming it has something to do with the sub-query that determines a rownumber?
Any help would be appreciated with this; my searches on google have yielded unhelpful results :(
If you wish to apply paging in MS Acces use this
SELECT *
FROM (
SELECT Top 5 sub.ClientCode
FROM (
SELECT TOP 15 tblClient.ClientCode
FROM tblClient
ORDER BY tblClient.ClientCode
) sub
ORDER BY sub.ClientCode DESC
) subOrdered
ORDER BY subOrdered.ClientCode
Where 15 is the StartPos + PageSize, and 5 is the PageSize.
EDIT to comment:
The error you are receiving, is because you are trying to reference a column name assign in the same level of the query, namely rownumber. If you were to change your query to:
SELECT *
FROM (
SELECT ClientCode,
(SELECT COUNT(c2.ClientCode)
FROM tblClient AS c2
WHERE c2.ClientCode <= c1.ClientCode) AS rownumber
FROM tblClient AS c1
)
WHERE rownumber BETWEEN 0 AND 15
It should not give you an error, but i dont think that this is the paging result you want.
See astander's answer for the original answer, but here's my final implementation that takes into account some ODBC parser rules (for the first 15 records after skipping 30):
SELECT *
FROM (
SELECT Top 15 -- = PageSize
*
FROM
(
SELECT TOP 45 -- = StartPos + PageSize
*
FROM tblClient
ORDER BY Client
) AS sub1
ORDER BY sub1.Client DESC
) AS clients
ORDER BY Client
The difference here is that I need the pagination to work when sorted by client name, and I need all columns (well, actually just a subset, but I sort that out in the outer-most query).
I use this SQL code to implement the pagination with Access
Select TOP Row_Per_Page * From [
Select TOP (TotRows - ((Page_Number - 1) * Row_Per_Page)
From SampleTable Order By ColumnName DESC
] Order By ColumnName ASC
I've published an article with some screenshots
on my blog
This is the simple method of pagination using OleDbDataAdapter and Datatable classes. I am using a different SQL command for simplicity.
Dim sSQL As String = "select Name, Id from Customer order by Id"
Dim pageNumber As Integer = 1
Dim nTop As Integer = 20
Dim nSkip As Integer = 0
Dim bContinue As Boolean = True
Dim dtData as new Datatable
Do While bContinue
dtData = GetData(sSQL, nTop, nSkip, ConnectionString)
nSkip = pageNumber * nTop
pageNumber = pageNumber + 1
bContinue = dtData.Rows.Count > 0
If bContinue Then
For Each dr As DataRow In dtData.Rows
'do your work here
Next
End If
Loop
Here is the GetData Function.
Private Function GetData(ByVal sql As String, ByVal RecordsToFetch As Integer, ByVal StartFrom As Integer, ByVal BackEndTableConnection As String) As DataTable
Dim dtResult As New DataTable
Try
Using conn As New OleDb.OleDbConnection(BackEndTableConnection)
conn.Open()
Using cmd As New OleDb.OleDbCommand
cmd.Connection = conn
cmd.CommandText = sql
Using da As New OleDb.OleDbDataAdapter(cmd)
If RecordsToFetch > 0 Then
da.Fill(StartFrom, RecordsToFetch, dtResult)
Else
da.Fill(dtResult)
End If
End Using
End Using
End Using
Catch ex As Exception
End Try
Return dtResult
End Function
The above codes will return 10 rows from the table Customer each time the loop operate till the end of file.
One easy way to use limit or get pagination working in access is to use ADODB library which support pagination for many DBs with same syntax. http://phplens.com/lens/adodb/docs-adodb.htm#ex8
Its easy to modify/override pager class to fetch required number of rows in array format then.
SELECT *
FROM BS_FOTOS AS TBL1
WHERE ((((select COUNT(ID) AS DD FROM BS_FOTOS AS TBL2 WHERE TBL2.ID<=TBL1.ID)) BETWEEN 10 AND 15 ));
Its result 10 to 15 records only.