I have generated a datatable by querying a large mysql-database.
This resulted in about 7000 rows. Now I would like to add a column that calculates the sum of column "QUANTITY" for every unique value in column "BREAK2" (like a 'group by').
Example:
BREAK2 | QUANTITY | COMPUTED
A 10 30
B 20 40
A 10 30
A 20 30
B 20 40
If I use
Dim dc As DataColumn = New DataColumn
dc.DataType = System.Type.GetType("System.Double")
dc.ColumnName = "SumQTY"
dt.Columns.Add(dc)
For Each dr As DataRow In DT.Rows
dr("SumQTY") = DT.Compute("Sum(QUANTITY)", "BREAK2 = '" & dr("BREAK2") & "'")
Next
it takes a long time (several minutes before I break it off).
When I query the same data with a querytool named BRIO, and I add a computed column with formula "Sum(QUANTITY,BREAK2)", it takes about 1 second to get the results.
What could be the difference between these methods?
Why is the VB.net code taking so long?
I have tried to use LINQ but same bad performance.
Thanks.
Nas
This seems to be pretty fast
Sub Main
Dim sw = new Stopwatch()
sw.Start()
Dim rnd = new Random()
Dim dt = new DataTable()
dt.Columns.Add("BREAK2", "".GetType())
dt.Columns.Add("QUANTITY", 0.GetType())
dt.Columns.Add("COMPUTED", 0.GetType())
for i = 0 to 7000
dt.Rows.Add("A", rnd.Next(0, 30))
dt.Rows.Add("B", rnd.Next(0, 30))
dt.Rows.Add("C", rnd.Next(0, 30))
dt.Rows.Add("D", rnd.Next(0, 30))
Next
Dim keys As List(Of String) = dt.AsEnumerable() _
.Select(Function(x) x("BREAK2").ToString()) _
.Distinct().ToList()
For Each k in keys
Dim listOfRow = dt.AsEnumerable() _
.Where(Function(x) x.Field(Of String)("BREAK2") = k).ToList()
Dim total = listOfRow.Sum(Function(t) t.Field(Of Integer)("QUANTITY"))
For Each row in listOfRow
row.SetField(Of Integer)("COMPUTED", total)
Next
Next
sw.Stop()
Console.WriteLine(sw.ElapsedMilliseconds)
End Sub
As I have said in my comment above, it is silly to recalculate the same value for thousands of time.
Using LINQ you could extract the unique values of the column BREAK2, execute the sum just one time for the subset of rows identified by the key and then set the same subset of rows with the total.
On my PC this takes at max 100 milliseconds.
Related
I have dgv which contains table like follow that check number of item. And now I need to calculate average after key in data.
ID
Part No
Item 1
Item 2
Item 3
Average
P01
Top
11
14
12
12.3
P02
Middle
12
15
11
12
P03
Bottom
10
13
16
13
I have create coding like this and it shows no error.
For Each c As DataColumn In dgv1.Columns
Dim j, sum As Integer
If dgv1.Rows(0).Item(c).Contains(Data & "" & (j)) Then
For i = 0 To dgv1.Rows.Count - 1
Try
sum += Convert.ToInt32(dgv1.Rows(i).Cells(j).Value)
Catch ex As Exception
End Try
Next
dgv1.Item("Average", curRow).Value = sum / (j)
End If
Next
But when run the system, it stated unable to cast object of type 'system.windows.form.datagridviewtextboxcolumn' to type 'system.data.datacolumn'
what does it mean because if i use datagridviewcolumn for datacolumn also cannot run. I am very new on VB.net and I need someone help on this.Is there a better way or reference for this? Thanks
It is unclear how the “average” column is inserted into the grid. I would assume that the “Average” column is NOT getting returned from the DB and is added by your code.
In that case a simple solution is to add the “Average” column to the DataTable returned from the DB. Then set that columns “Expression” string to calculate the average you are wanting.
To show a full example of this, I will assume that when the code gets the data from the data base, that it returns a DataTable. If this is the case then all we need to do is add the “expression” column to that DataTable. Adding the “expression” column to the DataTable may look something like...
Private Sub AddExpressionColumn(GridTable As DataTable)
Dim col = New DataColumn("Average")
col.DataType = GetType(Decimal)
col.Expression = "(Item1 + Item2 + Item3) / 3"
GridTable.Columns.Add(col)
End Sub
Above, the code takes a DataTable with existing columns named “Item1”, “Item2” and “Item3” and adds those column values together then divides by three (3). Now, if the user “changes” one of the “Item” values, then, the “Average” value will update automatically.
To show a full example, create a new VB winforms project, drop a DataGridView onto the form, and finally paste the code below to see this in action. The picture below shows the added “Average” column in light green.
Dim rand As Random = New Random()
Private Sub Form1_Load(sender As Object, e As EventArgs) Handles MyBase.Load
Dim GridTable As DataTable = GetDataFromDB()
AddExpressionColumn(GridTable)
dataGridView1.DataSource = GridTable
dataGridView1.Columns("Average").DefaultCellStyle.Format = "0.0"
End Sub
Private Sub AddExpressionColumn(GridTable As DataTable)
Dim col = New DataColumn("Average")
col.DataType = GetType(Decimal)
col.Expression = "(Item1 + Item2 + Item3) / 3"
GridTable.Columns.Add(col)
End Sub
Private Function GetDataFromDB() As DataTable
Dim dt = New DataTable()
dt.Columns.Add("ID", GetType(String))
dt.Columns.Add("Part No.", GetType(String))
dt.Columns.Add("Item1", GetType(Int32))
dt.Columns.Add("Item2", GetType(Int32))
dt.Columns.Add("Item3", GetType(Int32))
dt.Rows.Add("PO1", "Top", 11, 14, 12)
dt.Rows.Add("PO2", "Middle", 12, 15, 11)
dt.Rows.Add("PO3", "Bottom", 10, 13, 16)
For i = 4 To 10
dt.Rows.Add("PO" + i.ToString(), "Part" + i.ToString(), rand.Next(1, 50), rand.Next(1, 50), rand.Next(1, 50))
Next
Return dt
End Function
I hope this makes sense and helps.
Why make it so complicated.
For i = 0 To dgv1.Rows.Count - 1
Dim Total as decimal = 0
for x = 3 to dgv1.Columns.Count - 2
Total = Total + val(dgv1(x,i).Value)
Next
Dim j as integer = dgv1.Columns.Count - 3
dgv1.Item("Average", i).Value = Total/j
Next
Do test it and let me know the result
I would like for my code to express the sum of random numbers generated to be in a range. Between 120 and 235.
What's the best way to do that without changing my code too much?
I'm positive it needs to create 2 Dims and an if else statement, but I can't word it properly.
I'm using Visual Studio 2017
Public Class Form1
Private Sub Button1_Click(ByVal sender As Object, e As EventArgs) Handles Button1.Click
Randomize()
TextBox1.Text = Rand(1, 100)
TextBox2.Text = Rand(Long.Parse(TextBox1.Text), 100)
TextBox3.Text = Rand(Long.Parse(TextBox2.Text), 100)
TextBox4.Text = Rand(Long.Parse(TextBox3.Text), 100)
TextBox5.Text = Rand(Long.Parse(TextBox4.Text), 100)
TextBox6.Text = Rand(Long.Parse(TextBox5.Text), 100)
End Sub
Private Function Rand(v As Long) As String
Throw New NotImplementedException()
End Function
Private Function Rand(ByVal Low As Long, ByVal High As Long) As Long
Rand = Int((High - Low + 1) * Rnd()) + Low
End Function
End Class
I'ld suggest to use the .Net Random class to generate random numbers. It's also simpler to use.
Find the first random number between 0 and the Minimum value, then the second random number will be in range: randomMin(MinValue) => (MinValue - randomMin, MaxValue - randomMin):
randomMin = rnd1.Next(Min + 1)
randomMax = rnd2.Next(Min - randomMin, Max - randomMin + 1)
result = randomMin + randomMax
To remember that, in the Random class, the upper limit is exclusive, so we need to add 1 to the Max value to include it in the range of random values.
Make a sample test:
(These code samples suppose that the VB.Net version in use is at least V.14, VS 2015+)
Private rnd1 As Random = New Random()
Private rnd2 As Random = New Random()
'(...)
Dim Min As Integer = 120
Dim Max As Integer = 235
For i = 0 To 100
Dim randomValues = GetRandomNumbersInRange(Min, Max)
Console.WriteLine($"Random Min: {randomValues.rndMin} Random Max {randomValues.rndMax}")
Console.WriteLine($"Sum: {randomValues.rndMin + randomValues.rndMax}")
Next
'(...)
Private Function GetRandomNumbersInRange(Min As Integer, Max As Integer) As
(rndMin As Integer, rndMax As Integer)
Dim randomMin As Integer = rnd1.Next(Min + 1)
Return (randomMin, rnd2.Next(Min - randomMin, Max - randomMin + 1))
End Function
If you want the method to directly return the sum, you could change the method return type like this:
Dim Min As Integer = 120
Dim Max As Integer = 235
For i = 0 To 100
Console.WriteLine(GetSumRandomNumbersInRange(Min, Max))
Next
'(...)
Private Function GetSumRandomNumbersInRange(Min As Integer, Max As Integer) As Integer
Dim randomMin As Integer = rnd1.Next(Min + 1)
Return randomMin + rnd2.Next(Min - randomMin, Max - randomMin + 1)
End Function
The random numbers could also be selected with:
randomMid(MaxValue - MinValue) => (MinValue, MaxValue - randomMid)
It this case, possibly implemented as:
Private Function GetSumRandomNumbersInRange2(Min As Integer, Max As Integer) As Integer
Dim randomFirst As Integer = rnd1.Next(Max - Min + 1)
Return randomFirst + rnd2.Next(Min, Max - randomFirst + 1)
End Function
Here is the algorithm I would use to accomplish this task:
Generate a single random number in the range 120 to 235. That will be the final sum.
Generate six random numbers in the range 0.0 to 1.0.
Normalise those six numbers, i.e. divide each by the sum of all six.
Multiply each of the six normalised numbers by the original random number.
Round each of the six results.
You now have six random numbers with a sum in the desired range. You may just need to add or subtract 1 to one of the values in case rounding has pushed the sum outside the range. You can choose one of the numbers at random to adjust.
'Create a random number generator.
Dim rng As New Random
'Create a random number in the desired range for the final sum.
Dim sum = rng.Next(120, 235 + 1)
'Generate six proportional values as fractions of 1.0.
Dim proportions = Enumerable.Range(1, 6).Select(Function(n) rng.NextDouble()).ToArray()
'Get the sum of all the proportional values.
Dim proportionsSum = proportions.Sum()
'Normalise the proportional values so that they sum to 1.0
proportions = Array.ConvertAll(proportions, Function(r) r / proportionsSum)
'Break the final sum up into the specified proportions.
Dim numbers = Array.ConvertAll(proportions, Function(r) CInt(Math.Round(r * sum)))
'Adjust as required if rounding has pushed the sum below the minimum value.
Do While numbers.Sum() < 120
'Get a random element index.
Dim index = rng.Next(0, numbers.Length)
'Increment the element at that index.
numbers(index) = numbers(index) + 1
Loop
'Adjust as required if rounding has pushed the sum above the maximum value.
Do While numbers.Sum() > 235
'Get a random element index.
Dim index = rng.Next(0, numbers.Length)
'Decrement the element at that index.
numbers(index) = numbers(index) - 1
Loop
'The numbers array now contains six random values with a sum in the range 120 to 235.
VB.NET Winforms
I need to compare 2 datatables (that are each within their own dataset) and conclude with a judgement of OK or NG, one is a reference table, the other is raw data from a machine. One table would have one row with a QTY, the other table would have multiple rows to match that QTY, hopefully. The purpose here is verify torque values on fasteners. The first table here is the data I would get as what would be required for the torques to pass. Let's call this table dtTorquesRequired:
NINDEX NQTY_REQ NMIN NMAX
7 1 33.0 59.0
8 4 33.0 59.0
9 2 4.5 7.5
12 2 4.5 7.5
NINDEX will be the common column between the 2 datasets.
Let's use the last row as an example. The last row tells me that I need to have 2 torques from NINDEX 12 with a value between 4.5 - 7.5.
This is the data that I have to judge. Let's call this table dtTorquesPerformed:
NINDEX NTORQUE_NO NTORQUE_STATUS NTORQUE_VALUE
7 1 1 42.56
8 1 1 42.22
8 2 1 42.49
8 3 1 42.10
8 4 1 42.37
9 1 1 7.01
9 2 1 7.10
12 1 1 5.68
12 2 1 5.81
The judgement needs to have NINDEX match between the 2 tables, NTORQUE_STATUS must be 1, and NTORQUE_VALEU must be between NMIN and NMAX in dtTorquesRequired.
Now, I am currently doing this in my application using only SQL row count results, but it is slow. I am doing it via multiple chatty calls instead of a chunky call to the DB, and then work with the resulting datasets.
This is how I am currently doing it, but it is slow. I want to do all the processing locally instead of using SQL:
Private Function TorqueJudgement(ckanban As String) As Boolean
Dim ccode As String
Dim cline As String
Dim dsTorquesRequired As New DataSet
Dim dtTorquesRequired As New DataTable
Dim AreAllTorquesGood As Boolean = False
Dim BadTorqueCount As Int16 = 0
ccode = ckanban.Substring(0, 5)
cline = ckanban.Substring(5, 2)
SQL.ExecQuery("SELECT ....") ''GETS DTTORQUESREQUIRED
dsTorquesRequired = SQL.SQLDS
dtTorquesRequired = dsTorquesRequired.Tables(0)
dtTorquesRequired.Columns.Add("PassFail")
Dim rc As Int16 = -1 '//
For Each dr As DataRow In dsTorquesRequired.Tables(0).Rows
rc += 1
Dim nindex As Int16 = dr("NINDEX")
Dim qtyrequired As Int16 = dr("NQTY_REQ")
Dim nmin As Decimal = dr("NMIN")
Dim nmax As Decimal = dr("NMAX")
SQL.AddParam("#CKANBAN", ckanban)
SQL.AddParam("#NINDEX", nindex)
SQL.AddParam("#QTY_REQ", qtyrequired)
SQL.AddParam("#NMIN", nmin)
SQL.AddParam("#NMAX", nmax)
SQL.ExecQuery("SELECT ...") ''Searches for rows that match dtTorquesRequired
dtTorquesRequired.Rows(rc)("PassFail") = SQL.RecordCount
If Not dtTorquesRequired.Rows(rc)("NQTY_REQ") <= dtTorquesRequired.Rows(rc)("PassFail") Then
BadTorqueCount += 1
End If
Next
If BadTorqueCount > 0 Then
AreAllTorquesGood = False
ElseIf BadTorqueCount = 0 Then
AreAllTorquesGood = True
End If
Return AreAllTorquesGood
End Function
How can I query a datatable object locally using VB.NET without using SQL
Thanks in advance
You can use DataView objects to "query" DataTables:
Dim dtb As New DataTable
dtb.Columns.Add("Col0")
dtb.Columns.Add("Col1")
dtb.Columns.Add("Col2")
dtb.Columns.Add("Col3")
dtb.Rows.Add("R0C0", "R0C1", "A", "A")
dtb.Rows.Add("R1C0", "R1C1", "A", "A")
dtb.Rows.Add("R2C0", "R2C1", "A", "B")
dtb.Rows.Add("R3C0", "R3C1", "B", "B")
Dim dvw As New DataView(dtb)
dvw.RowFilter = "Col0>'R1C0'" 'WHERE
dvw.Sort = "Col2 DESC" 'ORDER BY
For Each drv As DataRowView In dvw
Dim drw As DataRow = drv.Row
Console.WriteLine(drw("Col0") & " " & drw("Col1") & " " & drw("Col2") & " " & drw("Col3"))
Next
Dim dtbDistinct As DataTable = dtb.DefaultView.ToTable(True, {"Col2", "Col3"}) 'DISTINCT
For Each drw As DataRow In dtbDistinct.Rows
Console.WriteLine(drw("Col2") & " " & drw("Col3"))
Next
Console.ReadKey()
i am replacing old binary search functions with linq queries in my project.
i cant figure out why linq queries is so slow compare to the binary search method (must have been doing something wrong).
the execute of this part of the program takes 2-3 seconds with binary search compare to almost 60 sec with the linq searches i have written.
basically what i am trying to do is return index of dataset record.
i am looping trough datatable with 5000 records and passing the primary key of the datatable to the linq query \ binary search function in order to search that primary key in a global dataset and return the index of the record.
i must have been doing something wrong i will be greatful for your advice how to write this query more efficent.
the 2 functions of the linq query:
Public Function existInDs(ByVal spcall As String) As Boolean
'check if exist
Dim spIsExist As Boolean = db.ReadDataSet.Tables(0).AsEnumerable().Any(Function(x) x.Field(Of String)("SpCall") = spcall)
Return spIsExist
End Function
Private Function getIndexDsWithLinq(ByVal sp As Integer) As Integer
Dim SpCall = CType(sp, String)
If existInDs(SpCall) = False Then
Return -1
Else
Dim index As Integer = db.ReadDataSet.Tables(0).AsEnumerable().Where(Function(x) x.Field(Of String)("SpCall") = SpCall) _
.Select(Function(x) db.ReadDataSet.Tables(0).Rows.IndexOf(x)).Distinct.SingleOrDefault()
Return index
End If
End Function
i have also tried that:
Private Function getIndexDsWithLinq(ByVal sp As Integer) As Integer
Dim SpCall = CType(sp, String)
Dim columnIndex As Integer
Dim dr() As DataRow
dr = Me.db.ReadDataSet.Tables(0).Select("SpCall='" & SpCall & "'")
If dr.Length > 0 Then
columnIndex = Me.db.ReadDataSet.Tables(0).AsEnumerable().Where(Function(x) x.Field(Of String)("SpCall") = sp AndAlso x.Field(Of Date?)("Changed").HasValue).Select(Function(x) Me.db.ReadDataSet.Tables(0).Rows.IndexOf(x)).Distinct.SingleOrDefault()
Return columnIndex
Else
Return -1
End If
End Function
that is the(classic) binary search function:
Private Function Searchbinary(ByVal SpCall As Integer) As Integer
Dim lower As Long = 0
Dim upper As Long = db.ReadDataSet.Tables(0).Rows.Count - 1
Dim middle As Long
Dim rowNum As Integer
Do
middle = (lower + upper) / 2
If SpCall < CType(db.ReadDataSet.Tables(0).Rows(middle).Item("SpCall"), Integer) Then
upper = middle - 1
Else
lower = middle + 1
End If
Loop While CType(db.ReadDataSet.Tables(0).Rows(middle).Item("SpCall"), Integer) <> SpCall And lower <= upper
If CType(db.ReadDataSet.Tables(0).Rows(middle).Item("SpCall"), Integer) = SpCall Then
rowNum = middle
Else
rowNum = -1
End If
Return rowNum
End Function
Seems, you want to get first occurence of matching data and return its rowIndex.
The simplest way to achieve that using Linq is:
Dim dr As DataRow = Me.db.ReadDataSet.Tables(0).AsEnumerable() _
.Where(Function(x) x.Field(Of String)("SpCall") = SpCall).FirstOrDefault()
Dim rowIndex As Integer = Me.db.ReadDataSet.Tables(0).Rows().IndexOf(dr)
Please, check MSDN documentation
I tested code on below example:
Sub Main
Dim dt As DataTable = CreateDt()
Dim dr As DataRow = dt.AsEnumerable() _
.Where(Function(x) x.Field(Of String)("SpCall") = 5).FirstOrDefault()
Dim rowIndex As Integer = dt.Rows.IndexOf(dr)
Console.WriteLine(rowIndex)
End Sub
'below function is created only dor testing purposes
Public Function CreateDt() As DataTable
Dim dt As DataTable = New DataTable()
dt.Columns().Add(New DataColumn("SpCall", Type.GetType("System.String")))
For i As Integer = 1 To 50000
Dim randomValue As Integer = CInt(Math.Floor((50 - 1 + 1) * Microsoft.VisualBasic.Rnd())) + 1
dt.Rows.Add(New Object(){randomValue})
Next
Return dt
End Function
It's executed in less then 1 sec.
so to summery all the discussion, first as
#Joachim said - "if you know that the values are sorted, binary search will be faster than Linq", i searched that topic over the web and also here in stackoverflow and also measured processing time of binary search vs few variations of LINQ queries/Lambda expressions (you can see comparing in comments) , my final conclusion is that #Joachim is right. lambda expressions are the best solution when there are multiple conditions and binary search is the best solution when there are no conditions and values are sorted (for minimum process time).
How can I limit the number of rows in Datagridview column to lets say 50 record and place the other 50 into the next column on the same Datagridview? In my vb.net project I can't be scrolling up and down.
The Datagridview pull the data from an excel sheet.
Appreciate your help
Try this.
dataGridView1.RowCount = 50; //Let say we have 50 rows
if (dataGridView1.Rows.Count >= 50)
{
//TODO:
}
I assumed this as single column table ..
dim tbl as DataTable '--------> this is your displayed table
dim nLimit as Integer = 15 '----- this is your limit as you want
dim dc as DataColumn
'get column name
dim sCol as String = tbl.Columns(0).Name
dim nRow as Integer = tbl.rows.count
'how much column ?
dim n as Integer = nRow / nLimit
if n * nLimit < nRow then n+= 1
for x as Integer = 1 to n
dc = New DataColumn
dc.DataType = System.Type.GetType("System.String")
dc.Caption = sCol & n.ToString
dc.ColumnName = sCol & n.ToString
tbl.Columns.Add(dc)
next
Then you can show it ...