Out Of Memory Exception - unmanaged memory - vb.net

Question

I am working on a web scraper, and it gernally works quite well. It will go through thousands of pages on most sites and complete sucessfully with no issues.
On a few sites, I am repeatedly seeing the same issue.
Insufficient memory to continue the execution of the program.
Edit:
I used perfmon to determine that the leak is happening in unmanaged memory.
I know because "private bytes" keeps increasing as the program runs, while bytes in all heaps stays stable.
(actually, it goes up and down, but gradually climbs. it usually runs out of memory in the code section i listed above, but i dont think that section is the cause, but rather a likely first victim because it uses a lot of memory... i think it releases it afterwards though)
Edit 2:
I followed the directions on this site:
http://www.codeproject.com/Articles/42721/Best-Practices-No-5-Detecting-NET-application-memo
and i used debugDiag to inspect the program.
After analyzing the data, debug diag told me what was responsible for the leak:
jscript.dll is responsible for 1.10 GBytes worth of outstanding allocations. The following are the top 2 memory consuming functions:
jscript!Parser::GenerateCode+167: 498.19 MBytes worth of outstanding allocations.
jscript!NoRelAlloc::PvAlloc+96: 292.99 MBytes worth of outstanding allocations.
I am not referencing jscript.dll in my application, it must be being used by the web browser controls which I am using.
System.Windows.Forms.WebBrowser
Thats my guess, at least.
I am also getting a message box that pops up with the title "Message From webpage" that says something to the effect of "out of memory at line X."
So, i figured that i could just dispose of the webbrowser objects and get my memory back - so i added a button with the following code:
Me.wbMain.Dispose() 'dispose all of thwe web-browsers
frmDebugger.wbDebugMain.Dispose()
Me.WBNewWin.Dispose()
GC.Collect() 'just for the heck of it
So, after running it for awhile, i stopped scraping and clicked my new button... it didnt make any difference at all. I was watching the total "Private Bytes" in perfmon, and it didn't even move.
Any ideas, anyone?
Edit 3:
I have tried a bunch of the recommended solutions, none of them seem to be working.
Someone suggested that it may be due to images not being cleared from the cache, but i disabled images from loading, so i know that is not the problem.
I also heard that IE7 had an issue, and that upgrading to IE8 would resolve it. I have IE8 and it still leaks memory.
Someone suggested that minimizing the form with the webbrowser control would release some memory. I tried, and it does not make a difference.
I have also been told that i should not expect the memory use to just drop, as i will have to wait for the garbage collector. It is not a leak in managed code, so GC.Collect() wont do anything. It is in unmanaged memory. Apparently the javascript functionality uses different memory, and theres no manual way to force a collection. But its getting to the point where it crashes, so obviously there is a problem.
I am adding a bounty of 50 to this question, and i will award it to anyone who helps me solve the leak. I wanted to try this solution:
http://www.codeproject.com/Questions/322884/WPF-WebBrowser-control-vs-Internet-Explorer-browse
but i am unable to figure out what the vb.net equivalent would be. I have tried online converters, and they error when converting this code (though they work fine for other code i have converted in the past)
If i am unable to solve the leak, i will award it to anyone who converts the page i mentioned above from c# to vb.net.
My fallback plan is to create a separate application that only contains the webbrowser, and communicate with that process, until it runs low on memory, at which point i will restart it (memory is releasd when i clsoe my application completely). This is far from ideal for my application, as the webbrowser is woven pretty tightly into my project.
Edit 4
I tried to implement the javascript injection suggested - here is my code:
(I fire it just before navigating to a new page)
Public Shared Sub Clean_JS(ByRef wb As System.Windows.Forms.WebBrowser)
Dim args As Object() = {"document.body"}
Dim head As HtmlElement = wb.Document.GetElementsByTagName("head")(0)
Dim scriptEl0 As HtmlElement = wb.Document.CreateElement("script")
Dim element0 As mshtml.IHTMLScriptElement = DirectCast(scriptEl0.DomElement, mshtml.IHTMLScriptElement)
element0.text = "function ReleaseHandler() {" + vbCrLf + " var EvtMgr = (function() {" + vbCrLf + " var listenerMap = {};" + vbCrLf + " " + vbCrLf + " // Public interface" + vbCrLf + " return {" + vbCrLf + " addListener: function(evtName, node, handler) {" + vbCrLf + " node[""on"" + evtName] = handler;" + vbCrLf + " var eventList = listenerMap[evtName];" + vbCrLf + " if (!eventList) {" + vbCrLf + " eventList = listenerMap[evtName] = [];" + vbCrLf + " }" + vbCrLf + " eventList.push(node);" + vbCrLf + " }," + vbCrLf + " " + vbCrLf + " removeAllListeners: function() {" + vbCrLf + " for (var evtName in listenerMap) {" + vbCrLf + " var nodeList = listenerMap[evtName];" + vbCrLf + " for (var i = 0, node; node = nodeList[i]; i++) {" + vbCrLf + " node[""on"" + evtName] = null;" + vbCrLf + " }" + vbCrLf + " }" + vbCrLf + " }" + vbCrLf + " }" + vbCrLf + " })();" + vbCrLf + " }"
head.AppendChild(scriptEl0)
Dim scriptEl1 As HtmlElement = wb.Document.CreateElement("script")
Dim element1 As mshtml.IHTMLScriptElement = DirectCast(scriptEl1.DomElement, mshtml.IHTMLScriptElement)
element1.text = "function ReleaseHandler() {" + vbCrLf + " var EvtMgr = (function() {" + vbCrLf + " var listenerMap = {};" + vbCrLf + " " + vbCrLf + " // Public interface" + vbCrLf + " return {" + vbCrLf + " addListener: function(evtName, node, handler) {" + vbCrLf + " node[""on"" + evtName] = handler;" + vbCrLf + " var eventList = listenerMap[evtName];" + vbCrLf + " if (!eventList) {" + vbCrLf + " eventList = listenerMap[evtName] = [];" + vbCrLf + " }" + vbCrLf + " eventList.push(node);" + vbCrLf + " }," + vbCrLf + " " + vbCrLf + " removeAllListeners: function() {" + vbCrLf + " for (var evtName in listenerMap) {" + vbCrLf + " var nodeList = listenerMap[evtName];" + vbCrLf + " for (var i = 0, node; node = nodeList[i]; i++) {" + vbCrLf + " node[""on"" + evtName] = null;" + vbCrLf + " }" + vbCrLf + " }" + vbCrLf + " }" + vbCrLf + " }" + vbCrLf + " })();" + vbCrLf + " }"
head.AppendChild(scriptEl1)
wb.Document.InvokeScript("ReleaseHandler")
wb.Document.InvokeScript("purge", args)
End Sub
unfortunately, i am still seeing privaty bytes increasing in perfmon.
can anyone see any flaws in my logic? I am trying to implement this fix:
http://www.codeproject.com/Questions/322884/WPF-WebBrowser-control-vs-Internet-Explorer-browse
btw - i tested it using simple code such as this:
object[] args = {"my important message"};
webBrowser1.Document.InvokeScript("alert",args);
and this:
Dim head As HtmlElement = wb.Document.GetElementsByTagName("head")(0)
Dim scriptEl As HtmlElement = wb.Document.CreateElement("script")
Dim element As mshtml.IHTMLScriptElement = DirectCast(scriptEl.DomElement, mshtml.IHTMLScriptElement)
element.text = "function sayHello() { alert('hello') }"
head.AppendChild(scriptEl)
wb.Document.InvokeScript("sayHello")
and it showed the message in both test cases.
Curiously, when i tried to test the script injection by doing this:
Dim head As HtmlElement = wbMain.Document.GetElementsByTagName("head")(0)
Dim scriptEl As HtmlElement = wbMain.Document.CreateElement("script")
Dim element As mshtml.IHTMLScriptElement = DirectCast(scriptEl.DomElement, mshtml.IHTMLScriptElement)
element.text = "function sayHello() { alert('hello') }"
head.AppendChild(scriptEl)
wbMain.Document.InvokeScript("sayHello")
RTB_RawHTML.Text = "TEST" + vbCrLf + wbMain.DocumentText
I didnt see the injected code reflected in the text box - the only change i saw was the word "test" appearing (i run the code RTB_RawHTML.Text = wbMain.DocumentText when the pages finish loading from the documentCompleted event...)

Answer 1

The code in your referenced article is not C#, it is Javascript. I believe the idea would be to inject the JS into your HTML page so that it can run when the page unloads, which will clean out the existing JS events.
You can check out this article for adding JS to a page in your WebBrowser control:
http://www.codeproject.com/Articles/94777/Adding-a-Javascript-Block-Into-a-Form-Hosted-by-We
Dim scriptText As String =
<string>
function ReleaseHandler() {
var EvtMgr = (function() {
var listenerMap = {};
// Public interface
return {
addListener: function(evtName, node, handler) {
node["on" + evtName] = handler;
var eventList = listenerMap[evtName];
if (!eventList) {
eventList = listenerMap[evtName] = [];
}
eventList.push(node);
},
removeAllListeners: function() {
for (var evtName in listenerMap) {
var nodeList = listenerMap[evtName];
for (var i = 0, node; node = nodeList[i]; i++) {
node["on" + evtName] = null;
}
}
}
}
})();
}
function purge(d){
var a = d.attributes, i, l, n;
if (a) {
for (i = a.length - 1; i >= 0 ; i -= 1) {
n = a[i].name;
if (typeof d[n] === 'function') {
d[n] = null;
}
}
}
a = d.childNodes;
if (a) {
l = a.length;
for (i = 0; i < l; i += 1) {
purge(d.childNodes[i]);
}
}
}
<string>
Dim head As HtmlElement = webBrowser1.Document.GetElementsByTagName("head")(0)
Dim script As HtmlElement = webBrowser1.Document.CreateElement("script")
Dim domElement As IHTMLScriptElement = CType(script.DomElement, IHTMLScriptElement)
domElement.text = scriptText
head.AppendChild(script)
I've not tested this code (I'm not really sure how I'd go about doing that since you've offered no example code yourself)... this is more of a suggestion for how you might proceed. I've never tried to insert JS into a WebBrowser control, so I'm not quite sure how you'd go about executing it (since, in theory, the JS will have already executed after loading the page, thus your injected JS would be "late to the party").
You'll also need to find a way to wire-up the document so that it calls both of these functions when it unloads. The idea is to eliminate JS memory leaks by eliminating JS objects and events, so simply having the functions declared is insufficient. I've seen a lot of articles online discussing how the OnBeforeUnload event is broken in the WebBrowser control (it doesn't fire correctly), so you may have quite a bit of work cut out for you.

Answer 2

May be you can tried code for not saving the cookie to the user computer. Cause temporary item can make several issue to user computer