I have been creating multiple background threads to parse xml files and recreate new xml files. Now the problem I am having is that even though I use synclock on global variables, I will still at times get errors and I am sure that this is just the crude way of coding I am doing, but I was wondering if someone had a better option.
program flow =
access local folder and upload all files into list
strip each file into xml entries and put these entries into an arraylist
parse for specific values and enter these values into a database table
now create a thread and take the arraylist of entries and the thread will reparse
thread parses and creates a new xml file
main thread continues with another function and then goes and get a file from list
I will add some code to show problem areas but if I have declared global variable in use does the different threads overwrite that value in the variable causing contamination.
For Each g In resultsList
gXmlList.Add(g)
Next
Dim bgw As New BackgroundWorker
bgw.WorkerSupportsCancellation = True
AddHandler bgw.DoWork, New DoWorkEventHandler(AddressOf createXML)
AddHandler bgw.RunWorkerCompleted, AddressOf WorkComplete
threadlist.Add(bgw)
bgw.RunWorkerAsync()
Private Sub createXML()
num += 1
Dim file As String = Module1.infile
xmlfile = directoryPath & "\New" & dateTime.Now.ToUniversalTime.ToString("yyyyMMddhhmmss") & endExtension
Thread.Sleep(2000)
Dim doc As XmlDocument = New XmlDocument
**xwriter = New XmlTextWriter(xmlfile, Encoding.UTF8)** this is where ioexception error
xwriter.Formatting = Formatting.Indented
xwriter.Indentation = 2
xwriter.WriteStartDocument(True)
xwriter.WriteStartElement("Posts")
I have global variables through out the app and should I be locking each one and does this not make using threads then useless.
Dim j As Integer = 0
I believe your biggest problem is not knowing what features in .Net are thread safe. A list for example is not (a dictionary is). While you may get away with it you will eventually run into problems with locking, etc.
Your using classes and variables that are not thread safe. Any time you are working with threads you have to be Extremely careful with locking. To answer your question, yes, you have to lock and unlock everything you are working with unless the type / method specifically handles it for you.
There are a lot of multi threading (PLINQ for example) in .Net 4.0 which handle a lot of the "grunt work" for you. While you should learn and understand how to do thread safe code yourself it will give you a head start.
Try passing the data into the createXML() method. That may help isolate the code from other data being accessed. I would suggest reading up on threading and learning how to do it without a background worker.
Global variables are generally a bad idea. Given your VB code I'm guessing this is a carry over from the VB6 world for you. That's not in any way intended to be insulting, just trying to help advance your skills forward. Variable scope should be as confined as possible.
Another thought looking at your code is to learn how to use String.Format() when building strings / paths.
Simple manual thread in VB to get you started:
Dim bThread As New Threading.Thread(AddressOf createXML)
bThread.IsBackground = True
bThread.Start()
Well if you are having issues with thread locking then you can simply wrap your action in the following manor.
'This will need to be out of scope so that all threads have access to it
Dim readerWriterLock As New Threading.ReaderWriterLockSlim
readerWriterLock.EnterWriteLock()
xwriter = New XmlTextWriter(xmlfile, Encoding.UTF8)
'other logic
readerWriterLock.ExitWriteLock()
'anything reading from this would need to have the following
readerWriterLock.EnterReadLock()
'logic
readerWriterLock.ExitReadLock()
Try this and then if not successful post the exception message and any other information that you can.
Related
I am writing a program that writes data to a text file at different points in my code, for example in different subroutines, functions or at different parts of subroutines (being scattered around).
First, I Dim the file writer:
Dim CurrentHisWriter As System.IO.StreamWriter
I tell it where to write to:
CurrentHisWriter = New System.IO.StreamWriter("C:\ProgramData\Japanese Conjugation Helper\LastSearch.txt")
Then, I actually write things:
CurrentHisWriter.Writeline("thing to write")
The problem is that I have to change to a different subroutine and then keep on writing to a file, so I have to close the writer and then dim another one in another subroutine:
CurrentHisWriter.Close
NewSubroutine()
[NewSubroutine]:
Dim CurrentHisWriter As System.IO.StreamWriter
CurrentHisWriter = New System.IO.StreamWriter("C:\ProgramData\Japanese Conjugation Helper\LastSearch.txt")
But then when I do this, I gives me one of a couple errors:
The program is has an instance of the file running
Some thing to do with there being no object (I don't remember exactly)
What is a reliable way programming the writing to files without having to worry about closing the writer at every point I change subroutines. I'm not sure about how objects and instances work and so the only thing I can do now is make a catch loop around every single line that uses the "CurrentHisWriter.Writeline" but this isn't really working too.
I know my lack of knowledge in this doesn't help explain, but I tried my best.
The naive approach would be like:
Sub Main()
MethodA()
MethodB()
End Sub
Sub MethodA()
Log("Starting method A")
End Sub
Sub MethodB()
Log("Starting method B")
End Sub
Sub Log(message as String)
System.IO.File.AppendAllText("C:\temp\my.log", message)
End Sub
File.AppendAllText is pretty good at closing things off so you can subsequently write to it elsewhere
A better approach would be to have a class whose job it is to build this file, and it builds it all into a stringbuilder and then writes it once. Multiple of your methods use that class, build that file... The class can either implement some timed/periodic dumping of data to disk (if it's like logging, never ending, thousands of events per second.. but then perhaps you'd just use a logging framework rather than reinvent the wheel), or it has a write method that saves the rendering of it to disk
If there is another specialized application of your data at work here, for example if you're generating XML or JSON you should look at specific serialization approaches for those (wheels that have already been invented)
I use
FileOpen(1, "file.txt", OpenMode.Append)
Now you can write from any other subroutine
PrintLine(1, "text to write")
Until the file is closed
FileClose(1)
But maybe you could solve your problem this way:
Define CurrentHisWriter outside of subroutine as
Private CurrentHisWriter As System.IO.StreamWriter = ....
Then you won't have to close and reopen the writer, all your Subs and functions will have access to it.
I am currently making a vb program that i plan to make very big. I have a decent knowledge of visual basic but today i came across something i do not understand. Because of the huge size of my program , i decided to try and keep the program as organized as possible by putting specific subs in modules. These subs consist of httprequest , webbrowsers(control), webclients and alot of loops. In order to prevent these subs from lagging my main application i thread them using threading.thread and i start them from my main form. But this leads to two problems.
Problem 1: The threads cannot in any way interact with the main form.
Once the a httprequest or webclient collects the information from my desired website, i am trying to make it add the info to a listbox in my main form, So what i did is it typed
Msgbox("Info Sent")
form1.listbox1.items.add(String)
The first messagebox will show but although the code right under it runs, nothing is added to the first forms listbox.I am not using delegates to transfer the information, instead, although its not a good habit, i am using checkforillegalcrossovers.
Problem 2: Threading with a webbrowser.
Threading with a webbrowser using threading.thread also does not work because it causes an active x error. After looking it up i found that a solution was to use a single threaded apartment but this would not work because i may need multiple threads running off the same sub at once.
One solution that i have found to this problem is creating another form completely and setting it invisible, and since the form is its own thread i do not need to use threading.thread , but the problem comes when i am trying to create multiple threads, or else i can somehow dynamically create the threads and put the subs inside of it programically this method wont work And even if it does i feel that it is sloppy so i will leave this for one of two last resorts.
The other solution is the most simple one in which i just put all of the code in the main form, but if i keep on doing that form1 is gonna become huge and sloppy, doing this wont solve the webbrowser problem either and even when using regions i still feel that something that 1000+ lines deserves its own class.
There must be some solution out there that solves these problems. Any help would be appreciated, Thanks.
I checked my code for updating the progress bar, and using a single thread with synclock will NOT work. They way I make it work is perform the step of the pbar each time after a thread is started as I have limited total threads (say less than 5 threads). Thus, even the progress bar steps before the threads are finished, but it will not progress further before new threads started. It is not 100% accurate but it more or less telling the progress
'update the progress bar
some_form.PBar1.PerformStep()
' This while loop is to count the existing running thread,
' and determine whether new thread should start
While 1
Dim t2 = New System.Threading.Thread(Sub() WaitForPermission())
t2.Start()
t2.Join()
If proceed_gen Then
Exit While
End If
End While
'Start doing what I need to do
Dim t1 = SomeSub()
t1.Start()
'End of code, as VB doest not have thread.detach()
Correct me if I am wrong, but you probably have to use a background worker. I know this is annoying, but this is the limitation of VB.net.
Or, you can have something like this (pseudo code, not tested)
structure some_struct
'define the strings you want to update, and their status such that
'main() knows if you need to update the stuff to the form
' You can also put how many threads are running, and the status of each thread,
'such that the main knows if all threads are completed
end structure
sub your_sub()
'Manipulate the website, and update the data structure with
'proper stuff you need
end sub
sub main(){
dim t1 = New System.Threading.Thread(Sub() your_sub())
t1.start()
' I am listing only one threads here, but start as many as you want
'check if there are strings that you need to update to the form
while 1
'check if there are any stuff you want to update from the data structure.
' Make sure you use synclock on the data structure, so each thread won't fight each other on accessing the data struct
dim should_update as boolean = false
should_update = 'Something thatyou should implement to judge whether you should update the form.
'You can start a thread and join it so the thread won't fight with other threads for accessing the data structure
dim some_string as string
if should_update
some_string = 'You may also need a thread to join to get the stuff you need. Dim the string as an array if it is required.
'You can also try pass by ref
'if you need to use thread to access the data structure to know if you need to update the form
form1.listbox1.items.add(some_string )
end if
end while
end sub
This is an ugly solution, but it will help you do the job...
I have the following code which I am using to populate a ImageList from a SQLite database with images stored as blobs.
Public Sub populateImagesStyles()
ShoeImages1.Images.Clear()
StyleImagesLView.Items.Clear()
Dim s As SQLiteDataReader
Dim rcount As Integer = 0
dbLocalQuery = New SQLiteCommand("SELECT id, image FROM tblImages", dbLocal)
s = dbLocalQuery.ExecuteReader()
While s.Read()
rcount += 1
ShoeImages1.Images.Add(CStr(s("id")), byte2img(s("image")))
StyleImagesLView.Items.Add(CStr(s("id")), CStr(s("id")))
End While
s.Close()
Here is the byte2img function...
Public Function byte2img(ByVal imgByte As Byte()) As Image
Dim imgMemoryStream As System.IO.MemoryStream = New System.IO.MemoryStream(imgByte)
byte2img = Drawing.Image.FromStream(imgMemoryStream)
End Function
The database contains over 250 images and this process is completed twice on load to populate two different ImageList, because I need the images displayed at two different sizes.
When the process runs on loading the form, it causes the process to consume between 800MB and 1GB of system memory, unless I manually run the process again from an form control, which seems to trigger garbage collection.
Stepping through the loading process, it is clear that it is the byte2img process that is causing the memory usage to escalate - what is the best way to mitigate this?
Also, if anyone can think of a more efficient process to execute this, i'm all ears. The images have to be stored in the database file because I need to be able to just package the .db file and send it to a remote location at a moments notice, so I can't mess with folders with images.
All help appreciated.
You are creating a lot of memory streams without disposing of them. Try this:
Public Function byte2img(ByVal imgByte As Byte()) As Image
Dim img As Image
Try
Using ms As New MemoryStream(imgbyte)
img = Drawing.Image.FromStream(ms)
End Using ' auto dispose of the MS
Catch ex As Exception
' report possibly bad/missing imgByte()
' resulting in an error in either place
End Try
Return img
End Function
An imprecise way to detect this kind of thing is to watch the HANDLES count in TaskManager.
Ok, I've found a solution/workaround that seems to work - call the PopulateImageStyles sub when a user visits the specific TabPage the ImageList resides on.
For some arbitrary reason, when run this way (as above, when called on the form), the process never proceeds to consume more than 50-60 MB of working memory.
I'll add a Background Worker so that the process can execute without hanging the form.
My apologies in advance if this has already been answered, but every search I have done does not come close to what I need. Also, this is all pseudo code.
Here is the situation: I created a form (targeting DOT NET 3.5) that does a loop on a gridview recreating a class and runs the code. After the code runs, there is a local variable on the class that gets updated and allows me to use it and the process repeats. Something like this:
For x as Integer = 0 to Me.txtTextBox.Lines.Count - 1 'Can be in the hundreds
Dim objMyClass as MyClass = New MyClass(Me.DatagridView1.Rows(x).Cells(0).Value)
if objMyClass.Start() = True then
'Do my thing with objMyClass.LocalLongVariable
End If
Next
This works just fine, but takes literally days to complete. The last time I ran this it took like 6 days, 7 hours and 40 something minutes to complete and barely bumped the CPU usage.
So, now I want to use MulitThreading to run several of these instances at the same time. I have not been able to get this to work. Everything I try returns different values every time I run it (and it should not). I believe that the threads are accessing the local variable across other threads and are incrementing at will. And SyncLock locks up the entire program. I have also tried adding a custom event that fires when the process is completed and executes a delegate on the Main form, but that has not worked either.
Now, my question is simple: How can I run multiple threads using the same base class (passing a unique string variable) and have the local class variable produce the correct results back to the UI? (And, from what I have been reading, the BackgroundWorker class in not suitable for this many threads (like hundreds); correct me if I read it incorrectly please)
I am looking for something like:
Dim thrd(Me.txtTextBox.Lines.Count) as Thread
Dim objMyClass(Me.txtTextBox.Lines.Count) as MyClass
For x as Integer = 0 to Me.txtTextBox.Lines.Count - 1
thrd(x) = new Thread (Sub()
objMyClass(x) = New MyClass(Me.GridView1.Rows(x).Cells(0).Value
If objMyClass.Start() = True Then
'Do my stuff here (maybe call a delegate??)
End If
End)
thrd(x).IsBackground = True
thrd(x).Start()
Next
Any help/advice on how to proceed will be greatly appreciated. And, if you know of any examples of your suggestion, please post the code/link.
The solution was, in fact, Synclock. My issue was that I was locking the wrong object, objMyClass, instead of the current Me AND I was failing to use Monitor.PulseAll(). Also, I switched to using the ThreadPool.QueueUserWorkItem(AddressOf objMyClass, args) and also used SyncLock on my custom event raised when the thread completes. It's a whole lot easier!! Thanks!!
Threads a + b, (both are trying to delete files).
a gets called first, then b while a is still running.
b deletes the file successfully but a doesn't.
If I run a on its own, a's file deletes fine.
When I step through the code I can see that a's MultiAttemptFilename gets overwritten with b's.
I don't understand.
I have an ajax call pointing to a generic handler which passes the filename along with it.
In my handler I have the following code:
Dim Dc As New Document
Dim MyThread As New Thread(AddressOf Dc.DeleteFileMulitAttempt)
Dc.MulitAttemptFilename = Filename
MyThread.Start()
From my 'Document' class I'm calling the following:
#Region "Delete"
Public MulitAttemptFilename As String = ""
Public Sub DeleteFileMulitAttempt()
Dim TimeBetweenAttempts As Integer = 2000
Dim NumberOfAttempts As Integer = 60
Dim AttemptNumber As Integer = 0
Dim Success As Boolean = False
While (AttemptNumber < NumberOfAttempts)
Try
Success = (DeleteFile(MulitAttemptFilename) = "Ok")
Catch ex As Exception
Success = False
End Try
If (Success) Then Exit While
Thread.Sleep(TimeBetweenAttempts)
AttemptNumber += 1
End While
End If
End Sub
...
This is to handle cancelled/failed uploads as they don't always delete right away (server locks etc), hence the loop.
Am I missing something fundamental here?
It seems like you might be missing the fundamental concept of multi-threaded concurrency. There are books dedicated to this, and often sections of .NET books will address this issue. Here's just one article by Microsoft on the topic.
One short answer is you need to use VB's "lock" keyword. You create an object and you do roughly something like
lock(yourLockObject)
{
//any code that might access a shared resource like the variable
//MulitAttempFilename [sic] would go here.
}
I don't speak VB but it looks like you're making the one thing that really needs to be protected a global variable. Global data is pretty much a bad idea in any form and when it comes to multi-threading it's a really, really bad idea. You'll have to rewrite your code to protect access to the name of the file being deleted. While you're reading up on multi-threading you might also want to learn about thread pools.