I have a method and I want to run multiple threads to complete it faster.
Public Sub createMaster()
For intLoop = 0 To _MasterCount - 1 Step _Step
_MasterCollection(intLoop) = New Champion(_MasterIDCollection(intLoop), _RRM.returnChampionInformation(_MasterIDCollection(intLoop).Substring(0, _MasterIDCollection(intLoop).Length() - 1), "na"))
Next
End Sub
So, the method basically creates a collection for me. I've tried to do multiple threads to complete it faster, making a champion actually takes a second. Is there a way to make intLoop a specific variable? I tried making intloop = _Start where I increased _Start by one each time and it ended up making _Start all kinds of numbers. My main is like:
Dim thread1 As New System.Threading.Thread(AddressOf createMaster)
thread1.Start()
Dim thread2 As New System.Threading.Thread(AddressOf createMaster)
thread2.Start()
thread1.Join()
thread2.Join()
I tried using a For loop with the thread and it didn't seem to work either. Does anyone know how to make this work in any way?
You could convert your current code to use LINQ to map your _MasterIDCollection to Champion instances:
_MasterCollection = (
From id In _MasterIDCollection
Select New Champion(id, _RRM.returnChampionInformation(id.Substring(0, id.Length() - 1), "na"))
).ToList() ' or ToArray()?
LINQ is easily parallelizable by adding AsParallel, but you also need AsOrdered to maintain the order:
_MasterCollection = (
From id In _MasterIDCollection.AsParallel().AsOrdered()
Select New Champion(id, _RRM.returnChampionInformation(id.Substring(0, id.Length() - 1), "na"))
).ToList() ' or ToArray()?
By default, PLINQ will (I believe) run one thread per CPU core, but you can control that by adding WithDegreeOfParallelism. Whether this is worthwhile depends on the type of work being done (e.g. I/O-bound or CPU-bound):
_MasterCollection = (
From id In _MasterIDCollection.AsParallel().AsOrdered().WithDegreeOfParallelism(20)
Select New Champion(id, _RRM.returnChampionInformation(id.Substring(0, id.Length() - 1), "na"))
).ToList() ' or ToArray()?
This requires .NET 4+, a reference to System.Core and Using System.Linq. See the PLINQ docs for more info.
Related
What does my program do?
As at 05 May 2021
This program was developed with the language VB.Net, the .NET framework 4.8 and with Visual Studio 2019 CE. The point of this program is to run a rudimentary database. The view is similar to a classic Internet forum—there are threads, in the threads there are different numbers of postings and in each post there are different numbers of pictures and long texts. If the thread is selected using the ComboBox, all posts with their images and texts are displayed one below the other. When you click on a specific post, only its images are displayed. Since the database is only intended for the company's products, it was decided not to use categories (e.g. images vs. videos vs. offtopic because it doesn't make any sense) and sub-categories (e.g. electrical vs. wood products).
When the program is closed, you will be asked whether the data should be saved. (still in the beta version). These data are read in when the program is loaded. If images are not found, their paths will be displayed in a window.
The user also has the option of searching through all threads and viewing the results with various sorting options. In this case, only the posts found are listed in the ListBox, and here, too, the user can select individual posts and have them displayed enlarged.
The program reads in the user data when it starts. A user can log in and, depending on his role, has certain power to make decisions. A “normal” user can create threads and posts, but only an administrator or moderator can edit and delete posts; and block a user. If you are not logged in or if you are locked, you can only read threads and posts.
The number of contributions is counted for each user. In the future, it should be possible to give a user stars.
About the classes
There is the Form1.vb class, and three other important classes: Class_Forum, Class_Post and Class_Thread. There is also the Class_User class. If a new post is created, this instance of Class_Post is added to a List(of Class_Post), which is located in Class_Thread (“The thread knows which posts it has”). Class_Post has a member ‘Made_by’, which is an instance of the Class_Users (“Every post knows which user made it”). Class_Post contains the member ‚Bilder‘ (=Images), which is a List(Of Bitmap). That is, every instance of class_post has got a List(of Bitmap).
There are also several forms for 1) editing, deleting posts, 2) for blocking or unblocking users, 3) for displaying enlarged images, 4) for logging in, 5) for displaying when images are not loading found, 6) to open the thread, 7) to post.
When the program is started, i.e. when data is read in, the thread instances and post instances are created.
To do list:
1.)
I would like, however, that only the images are in the RAM, which belong to the thread selected with the combobox1. My question is: Do I have to dispose all unnecessary images and read them in again when required? Do we get that built in?
This is my Code to load the data from the formatted txt file. I have a feeling, somewhere in here, or immediately after here, I have to do something.
Private Sub Daten_laden()
Dim Pfad As String 'file path
Using OFD1 As New CommonOpenFileDialog
OFD1.Title = "Textdatei auswählen"
OFD1.Filters.Add(New CommonFileDialogFilter("Textdatei", ".txt"))
OFD1.InitialDirectory = Environment.GetFolderPath(Environment.SpecialFolder.MyDocuments)
Dim Result As CommonFileDialogResult
Me.Invoke(Sub() Result = OFD1.ShowDialog())
If Result = CommonFileDialogResult.Ok Then
Pfad = OFD1.FileName
Else
Return
End If
End Using
Dim Pruef_Anzahl_Posts_in_dem_Thread As Integer = 0 'Check number of posts in the thread
Dim Liste_mit_den_Pfaden As List(Of String) 'List with file paths
Dim Liste_mit_den_Bildern As List(Of System.Drawing.Bitmap)
'read all Text
Dim RAT() As String = System.IO.File.ReadAllLines(Pfad, System.Text.Encoding.UTF8)
Pfade_von_nicht_gefundenen_Bildern = New List(Of String) ' Paths of not found images
For i As Integer = 3 To RAT.Length - 2 Step 1
Liste_mit_allen_Threads.Add(New Class_Thread(RAT(i)))
Me.Invoke(Sub() RaiseEvent Es_wurde_ein_neuer_Thread_eroeffnet()) 'A new thread has been opened
Pruef_Anzahl_Posts_in_dem_Thread = CInt(RAT(i + 1))
For j As Integer = (i + 2) To RAT.Length - 2 Step 1
Liste_mit_den_Pfaden = New List(Of String)
Liste_mit_den_Bildern = New List(Of Bitmap)
Dim Index As Integer
Dim Die_ID_des_Nutzers_der_den_Post_erstellt_hat As ULong = CULng(RAT(j + 4)) 'The ID of the user who created the post
For u As Integer = 0 To alle_Nutzer_Liste.Count - 1 Step 1
If alle_Nutzer_Liste(u).ID = Die_ID_des_Nutzers_der_den_Post_erstellt_hat Then
Index = u
Exit For
End If
Next
Dim neuerPost As New Class_Post(RAT(j), RAT(j + 1), CUShort(RAT(j + 2)), Liste_mit_den_Bildern, CDate(RAT(j + 3)), Liste_mit_den_Pfaden, alle_Nutzer_Liste(Index))
'how many threads are there already
Dim wie_viele_Threads_gibt_es_bereits As Integer = Liste_mit_allen_Threads.Count
Liste_mit_allen_Threads(wie_viele_Threads_gibt_es_bereits - 1).Posts_in_diesem_Thread.Add(neuerPost)
' Set the index to the last possible one in the ComboBox. This causes the program to run into the Selected Index event and SI becomes the selected index.
Me.Invoke(Sub() ComboBox1.SelectedIndex = Liste_mit_allen_Threads.Count - 1)
j += 5
Do
Liste_mit_den_Pfaden.Add(RAT(j))
If System.IO.File.Exists(RAT(j)) Then
Liste_mit_den_Bildern.Add(New Bitmap(RAT(j)))
Else
Pfade_von_nicht_gefundenen_Bildern.Add(RAT(j))
End If
j += 1
Loop Until RAT(j) = "#" ' Marker: a post is over
If RAT(j + 1) = "" AndAlso RAT(j + 2) = "" Then ' A new thread is marked with 2 blank lines one below the other.
i = (j + 2)
Exit For
End If
Next
Next
Me.Invoke(Sub() alle_Posts_in_diesem_Thread_anzeigen()) 'show all posts in this thread
If Pfade_von_nicht_gefundenen_Bildern.Count > 0 Then ' In Case something went wrong
Using FBNG As New Form_Bild_nicht_gefunden
FBNG.Datei_anzeigen(Pfade_von_nicht_gefundenen_Bildern)
FBNG.ShowDialog()
End Using
End If
End Sub
On this image, the thread is being switched using the combobox which changes the variable SI which I use often. In this example, the thread Caucasian contains 1 post which contains 2 images. Which means, (still in this moment) I don't need images form shepherds thread
It is not wise to keep all data in RAM. When your database grows, there will be a time that there is too much data.
To overcome this problem, people invented databases: the data is saved on a disk, and only the data that you request is put into memory. Smart databases will keep important data and often used values in memory, to minimize the request time.
If I look at your program, it seems that you only have to change the display after operator input. Operator input is relatively slow: you're a good typist if you can type more than 3 character per second. Usually the response time after operator input is not a problem: if you get the data within half a second, no one will complain.
For a modern computer half a second is enough time to examine a million records. In your application a database won't be a problem.
So my advice would be: start using a database and load only the data that is needed right now, instead of reading all data at startup. Only if you experience long request times, consider to load data that you expect you will need very soon.
Alas, to use a database you will need to learn something new: at least how to structure a databases. If I look at your tables, I have the impression that you already mastered this. Furthermore you'll have to learn how to add / query / update / remove data. This is usually done using SQL or software that supports LINQ, like entity framework.
It seems to me that your queries are quite limited in number: you won't have hundreds of different queries. If you already know SQL, and you don't think you need to know entity framework in the near future, I would go for accessing the database using SQL.
If you don't know SQL very much, or if you need to do an awful lot of different queries, consider to access the database using LINQ. This requires entity framework.
If you haven't got a database already, my first shot would be to use SQLight: a database in one file, fast enough for your application.
If you hide properly that you use SQLight, migrating to a smarter database if the need arises won't require a big change in your application.
class Repository
{
public long AddPost(Post post) {...} // add Post to the database, returns the Id
public long AddUser(User user) {...}
...
// fetch all Posts of a User:
public User FetchUserWithHisPosts(int userId);
// fetch Posts of a User after a certain data:
public User FetchUserWithHisPosts(int userId, DateTime startDate);
...
}
A Repository is some kind of warehouse: all you know is that you can store items in it, and later retrieve them, even after your computer is restarted.
The Repository hides how it does this: the constructor might load everything in memory (like your current application), it could also be that the repository uses SQLight, or a smarter database, or even Entity Framework.
A good way to migrate would be to first translate your current application such that everyone only accesses the data using the Repository. The Repository accesses your "in-memory data" which is in a separate class that is loaded at startup.
Later you can change the repository such that it doesn't use the "in-memory data" anymore, but accesses the database: users of your repository won't have to change.
About loading pictures
No, you don't have to load all pictures at startup. It will be fast enough to load the pictures only when shown: after all, you won't be showing 1000 pictures on the screen at once.
As a picture uses a lot of memory, it is wise to Dispose() the picture as soon as you don't need it anymore:
Image GetImage(long imageId)
{
Repository repository = new Repository();
return repository.GetImageById(imageId);
}
void DisposeImage(Image image)
{
image.Dispose();
}
You hide how you Load the image, and how you free up memory after an Image is not needed anymore. This makes it easier to change this, might the need arise later. It also makes your code easier to read and to unit test.
Hello #Harald Coppoolse,
I thank you for your detailed and understandable answer. Admittedly, I have to look at and learn those SQL (Light) and Entity procedures.
For now, I've temporarily made sure that I only load the pictures that are shown and dispose all the others. I coded this as follows:
Private Sub ComboBox1_SelectedIndexChanged(sender As Object, e As EventArgs) Handles ComboBox1.SelectedIndexChanged
If ComboBox1.SelectedIndex <> (-1) Then
SI = ComboBox1.SelectedIndex
If Programm_fertig_geladen Then
Load_the_pictures_only_when_shown_and_free_up_memory_after_an_Image_is_not_needed()
End If
alle_Posts_in_diesem_Thread_anzeigen()
End If
End Sub
Private Sub Load_the_pictures_only_when_shown_and_free_up_memory_after_an_Image_is_not_needed()
For g As Integer = 0 To Liste_mit_allen_Threads.Count - 1 Step 1
If g <> SI Then 'Diese Bilder verwerfen
For i As Integer = 0 To Liste_mit_allen_Threads(g).Posts_in_diesem_Thread.Count - 1 Step 1
For j As Integer = 0 To Liste_mit_allen_Threads(g).Posts_in_diesem_Thread(i).Bilder.Count - 1 Step 1
Liste_mit_allen_Threads(g).Posts_in_diesem_Thread(i).Bilder(j).Dispose()
Liste_mit_allen_Threads(g).Posts_in_diesem_Thread(i).BoolArray(j) = False
Next
Dim CNT As Integer = Liste_mit_allen_Threads(g).Posts_in_diesem_Thread(i).Bilder.Count
For z As Integer = (CNT - 1) To 0 Step -1
Liste_mit_allen_Threads(g).Posts_in_diesem_Thread(i).Bilder.RemoveAt(z)
Next
Next
Else 'Diese Bilder laden
'für jeden Post
For i As Integer = 0 To Liste_mit_allen_Threads(g).Posts_in_diesem_Thread.Count - 1 Step 1
'die Bilder in dem Post
For j As Integer = 0 To Liste_mit_allen_Threads(g).Posts_in_diesem_Thread(i).Pfade_der_Bilder.Count - 1 Step 1
Dim neu_geladenes_Bild As New Bitmap(Liste_mit_allen_Threads(g).Posts_in_diesem_Thread(i).Pfade_der_Bilder(j))
If Liste_mit_allen_Threads(g).Posts_in_diesem_Thread(i).BoolArray(j) = False Then
Liste_mit_allen_Threads(g).Posts_in_diesem_Thread(i).Bilder.Add(neu_geladenes_Bild)
Liste_mit_allen_Threads(g).Posts_in_diesem_Thread(i).BoolArray(j) = True
End If
'End Using
Next
Next
End If
Next
End Sub
I built a Boolean array into the Class_Post, which has as many members as there are images and paths in the post. The purpose of this is to make sure that I remember which pictures are there and which are not.
When disposing of the images of a post, the respective array member is set to false.
When reloading the images of a post, the respective array member is set to True.
Dispose alone is not enough, because funnily enough, the place in the List(of Bitmap) remains occupied. You also have to completely delete the picture with RemoveAt, but in a backward running for loop, because the list gets smaller during the deletion. ;)
Using cannot not be used.
First timer here, so go easy on me. Is it theoretically possible to execute two OleDBDataAdapter.Fill calls on separate threads simultaneously - or is that fundamentally flawed?
Consider a form with 2 buttons and 2 datagridviews. Each button click launches a worker thread using an Async \ Await \ Task.Run pattern that calls a method to return a populated datatable and assigns it to one of the datagridviews. The .Fill in the first thread takes 30 seconds to complete. The .Fill in the second thread takes 1 second to complete. When launched individually, both buttons work as expected.
However, if I launch the first worker thread (30 seconds to Fill), then launch the second thread (1 second Fill), the second DataGridView is not populated until the first .Fill call completes. I would expect the second datagridview to populate in 1 second, and the first datagridview to populate ~30 seconds later.
I have duplicated this issue in my sample code with both the OleDBDataAdapter and the SqlDataAdapter. If I replace the long running query with a simple Thread.Sleep(30000), the second datagridview is populated right away. This leads me to be believe that it is not an issue with my design pattern, rather something specific to issuing the .Fill calls simultaneously.
Private Async Sub UltraButton1_Click(sender As Object, e As EventArgs) Handles UltraButton1.Click
Dim Args As New GetDataArguments
Args.ConnectionString = "some connection string"
Args.Query = "SELECT LongRunningQuery from Table"
Dim DT As DataTable = Await Task.Run(Function() FillDataTable(Args))
If DataGridView1.DataSource Is Nothing Then
DataGridView1.DataSource = DT
Else
CType(DataGridView1.DataSource, DataTable).Merge(DT)
End If
End Sub
Function FillDataTable(Args As GetDataArguments) As DataTable
Dim DS As New DataTable
Using Connection As New OleDbConnection(Args.ConnectionString)
Using DBCommand As New OleDbCommand(Args.Query, Connection)
Using DataAdapter As New OleDbDataAdapter(DBCommand)
DataAdapter.Fill(DS)
End Using
End Using
End Using
Return DS
End Function
Private Async Sub UltraButton2_Click(sender As Object, e As EventArgs) Handles UltraButton2.Click
Dim DS As DataTable = Await Task.Run(Function() LoadSecondDGV("1234"))
DataGridView2.DataSource = DS
End Sub
Function LoadSecondDGV(pnum As String) As DataTable
Dim DX As New DataTable
Using xConn As New OleDbConnection("some connection string")
Using DataAdapter As New OleDbDataAdapter("Select name from products where PNUM = """ & pnum & """", xConn)
DataAdapter.Fill(DX)
End Using
End Using
Return DX
End Function
This depends on what the data source is. Some data sources (like Excel) will only allow one connection at a time. Other data sources (like Access) will allow multiple connections, but actually fulfill the results in serial, such that you don't gain anything. Other data sources, like Sql Server, will allow the true parallel activity that you're looking for.
In this case, you mention that you also tried with an SqlDataAdapter, which indicates to me that you're talking to Sql Server, and this should be possible. What's probably going on here is that your first query is locking some of the data you need for the second query. You can get past this by changing your transaction isolation level or through careful use of the with (nolock) hint (the former option is strongly preferred).
One other thing to keep in mind is that this can only work if you're using a separate connection for each query, or if you've specifically enabled the Multiple Active Result Sets feature. It looks like you're using separate connection objects here, so you should be fine, but it's still something I thought was worth bringing up.
Finally, I need to comment on your FillDataTable() method. This method requires you to provide a completed Sql string, which practically forces you to write code that will be horribly vulnerable to sql injection attacks. Continuing to use the method as shown practically guarantees your app will get hacked, probably sooner rather than later. You need to modify this method so that it encourages you to use parameterized queries.
I have some very basic code that takes a list of strings and a new thread is created per string.
The thread then writes a file named with the string so should be unique but I get access violations.
I try the debugger option in VS 2010 and it doesn't give me the file access problems when I step through but it writes two lines per loop so I am assuming it still has a problem with running the thread twice
Code to call the thread:
For Each x In x_list
Dim trd as new Thread(DirectCast(Sub() tp(x), ThreadStart))
Next
Code for thread:
Private Sub tp(ByVal x_in As String)
Dim res_file As New StreamWriter("C:\result_" + x_in + ".txt", True)
For i = 1 to 5
res_file.WriteLine(DateTime.Now.ToString)
Next
res_file.Close()
End Sub
You're introducing an unnecessary Lambda here:
Sub() tp(x)
And then you're falling foul of the same well known issue from C# - that what you're capturing is the variable that keeps changing until it's set to the last value in the loop.1
Try instead just:
For Each x In x_list
Dim trd as new Thread(DirectCast(tp, ParameterizedThreadStart))
trd.Start(x)
'TODO - Does trd get stored anywhere?
Next
1 See e.g. Closing over the loop variable considered harmful which discusses how there's a change for C#5, but I don't think there's a corresponding change for VB.Net.
In a project I’ve recently taken over, there is a call to a function which does some calculations; this is called in a row, several times (between 1 and 10 times usually).
While dr.read ‘depending on a db call, loop 1 or more times
Dim calc As New CalcClass
Dim newDoStuff As New System.Threading.Thread(New System.Threading.ParameterizedThreadStart(AddressOf DoStuff))
newDoStuff.Start(calc)
End while
Private Sub DoStuff(ByVal calc As Object)
‘do something that takes between 5-10 seconds
End sub
In order to speed this up, i am trying to add asynchronous processing (see above example), this works in my code, all tasks are done at the same time, but what I don’t understand is how to then wait for all these threads to finish (there is no set amount of threads, it can be between 1 and 10 depending on some other data) before finishing up with a final task that needs to run after all tasks are completed.
Can anyone suggest a way to do this – I’m looking for an easy way to basically say “O.k, all tasks are finished at this point, call another task”
Cliffs
Several tasks need to run at the same time (between 1 and 10)
Each task takes several seconds
Code currently works - it does them all at the same time
Once all tasks (between 1-10) are
finished, fire off some other code (only when all tasks are finished) - stuck on best method to do the following
Put all your thread in an List
Dim threads As new List(Of System.Threading.Thread)
While dr.read ‘depending on a db call, loop 1 or more times
Dim calc As New CalcClass
Dim newDoStuff As New System.Threading.Thread(New System.Threading.ParameterizedThreadStart(AddressOf DoStuff))
threads.Add(newDoStuff)
newDoStuff.Start(calc)
End while
finally join all your threads
For Each thread In threads Do
thread.Join()
Next
The easiest way is to put all of your new threads into a list, then iterate over that list and call .Join() on each one. The join method blocks the current thread until the thread you are Joining completes:
Apologies if there are any syntactic errors in the following code- I don't have VB handy and my memory of the syntax is pretty rusty:
Dim threadList as New List(Of Thread)
While dr.read ‘depending on a db call, loop 1 or more times
Dim calc As New CalcClass
Dim newDoStuff As New System.Threading.Thread(New System.Threading.ParameterizedThreadStart(AddressOf DoStuff))
newDoStuff.Start(calc)
threadList.Add(newDoStuff)
End while
For Each t as Thread in ThreadList
t.Join
End For
With that said, I'd strongly encourage you to look into using the classes in the System.Threading.Tasks namespace, as that provides a much better paradigm than starting and managing your own threads.
I am having dataset with 5000 records.
I am reading them in the below 2 functions and writing to excel.
FillDataRows1(worksheet)
FillDataRows2(worksheet)
private sub FillDataRow1(byval ws as worksheet)
Dim rowpointer As Integer = 0
While rowpointer <= dsCostUsage.Tables(0).Rows.Count - 1
While colpointer <= dsCostUsage.Tables(0).Columns.Count - 1
str = dsCostUsage.Tables(0).Rows(rowpointer)(colpointer).ToString()
DirectCast(ws.Cells(row, column), Range).Value2 = item
colpointer += 1
End While
End While
End sub
private sub FillDataRow2(byval ws as worksheet)
Dim rowpointer As Integer = 1001
While rowpointer <= dsCostUsage.Tables(0).Rows.Count - 1
While colpointer <= dsCostUsage.Tables(0).Columns.Count - 1
str = dsCostUsage.Tables(0).Rows(rowpointer)(colpointer).ToString()
DirectCast(ws.Cells(row, column), Range).Value2 = item
colpointer += 1
End While
End While
End sub
I am reading 1000 records in the first function and remaining in the second function.
The problem is it is taking minimum 4 min to complete this process.
So, I decided ro use THREADING as
Dim t As New Thread(AddressOf FillDataRows1)
Dim t1 As New Thread(AddressOf FillDataRows2)
t.Start(worksheet)
t1.Start(worksheet)
t.Join()
t1.Join()
When I create thread also it is taking same time .
Am i doing anything wrong in creation of thread?
Is there any other way to improve the performance.
Every time you call an Excel function, it has to leave your application's thread, enter Excel's thread, execute the function, save the result, return to your application's thread, and retrieve the reesult. This is slow. Every individual ccall to Excel is slow. And a line such as:
Worksheets(1).Range("A1").Value += 1
is four separate calls to Excel, one for Worksheets, one for Range, and two for Value.
Do as much work as possible in your application. Read and write arrays of data from Excel using the Range.Value property over a large range.
Edit: I completely forgot to address the actual question.
Threading doesn't help because of the massive bottleneck in accessing Excel. I'm going to guess that the Excel API is only available on a single thread. So with multiple threads accessing Excel, each thread has to wait for the other to finish before it can talk to Excel.
If you pull a lump of data out of Excel, then have multiple threads process this data and create a new data table inside your application, you will see improvements.