Appication goto Freeze when downloading a webpage - httpwebrequest

i wrote a function for download a webpage : function like:
public string GetWebPage(string sURL)
{
System.Net.WebResponse objResponse = null;
System.Net.WebRequest objRequest = null;
System.IO.StreamReader objStreamReader = null;
string sResultPage = null;
try
{
objRequest = System.Net.HttpWebRequest.Create(sURL);
objResponse = objRequest.GetResponse();
objStreamReader = new System.IO.StreamReader(objResponse.GetResponseStream());
sResultPage = objStreamReader.ReadToEnd();
return sResultPage;
}
catch (Exception ex)
{
return "";
}
}
But my problem is that. when this function working at that time application goto freeze (not response) and that time my can't not do any thing. How can i solve this problem. when downloading at time user can do other thing in my application.

Welcome to the world of blocking IO.
Consider the following:
You want your program to download a web page and then return the first 10 letters it finds in the source html. Your code might look like this:
...
string page = GetWebPage("http://example.com"); // download web page
page = page.Substring(0, 10);
Console.WriteLine(page);
....
When your program calls GetWebPage(), it must WAIT for the web page to be fully downloaded before it can possibly try to call Substring() - else it may try to get the substring before it actually downloads the letters.
Now consider your program. You've got lots of code - maybe a GUI interface running - and it's all executing line by line one instruction at a time. When your code calls GetWebPage(), it can't possibly continue executing additional code until that request is fully finished. Your entire program is waiting on that request to finish.
The problem can be solved in a few different ways, and the best solution depends on exactly what you're doing with your code. Ideally, your code needs to execute asynchronously. c# has methods that can handle a lot of this for you, but one way or another, you're going to want to start some work - downloading the web page in your case - and then continue executing code until your main thread is notified that the webpage is fully downloaded. Then your main thread can begin parsing the return value.
I'm assuming that since you've asked this question, you are very new to threads and concurrency in general. You have a lot of work to do. Here are some resources for you to read up about threading and implementing concurrency in c#:
C# Thread Introduction
.NET Asynchronous IO Design

the best was is to use thread
new Thread(download).Start(url);
and if your download page size is large use chunk logic.
HttpWebRequest ObjHttpWebRequest = (HttpWebRequest)HttpWebRequest.Create(Convert.ToString(url));
ObjHttpWebRequest.AddRange(99204);
ObjHttpWebRequest.Timeout = Timeout.Infinite;
ObjHttpWebRequest.Method = "get";
HttpWebResponse ObjHttpWebResponse = (HttpWebResponse)ObjHttpWebRequest.GetResponse();
Stream ObjStream = ObjHttpWebResponse.GetResponseStream();
StreamReader ObjStreamReader = new StreamReader(ObjStream);
byte[] buffer = new byte[1224];
int length = 0;
while ((length = ObjStream.Read(buffer, 0, buffer.Length)) > 0)
{
downloaddata += Encoding.GetEncoding(936).GetString(buffer);

Related

Read a file from the cache in CEFSharp

I need to navigate to a web site that ultimately contains a .pdf file and I want to save that file locally. I am using CEFSharp to do this. The nature of this site is such that once the .pdf appears in the browser, it cannot be accessed again. For this reason, I was wondering if once you have a .pdf displayed in the browser, is there a way to access the source for that file in the cache?
I have tried implementing IDownloadHandler and that works, but you have to click the save button on the embedded .pdf. I am trying to get around that.
OK, here is how I got it to work. There is a function in CEFSharp that allows you to filter an incoming web response. Consequently, this gives you complete access to the incoming stream. My solution is a little on the dirty side and not particularly efficient, but it works for my situation. If anyone sees a better way, I am open for suggestions. There are two things I have to assume in order for my code to work.
GetResourceResponseFilter is called every time a new page is downloaded.
The PDF is that last thing to be downloaded during the navigation process.
Start with the CEF Minimal Example found here : https://github.com/cefsharp/CefSharp.MinimalExample
I used the WinForms version. Implement the IRequestHandler and IResponseFilter in the form definition as follows:
public partial class BrowserForm : Form, IRequestHandler, IResponseFilter
{
public readonly ChromiumWebBrowser browser;
public BrowserForm(string url)
{
InitializeComponent();
browser = new ChromiumWebBrowser(url)
{
Dock = DockStyle.Fill,
};
toolStripContainer.ContentPanel.Controls.Add(browser);
browser.BrowserSettings.FileAccessFromFileUrls = CefState.Enabled;
browser.BrowserSettings.UniversalAccessFromFileUrls = CefState.Enabled;
browser.BrowserSettings.WebSecurity = CefState.Disabled;
browser.BrowserSettings.Javascript = CefState.Enabled;
browser.LoadingStateChanged += OnLoadingStateChanged;
browser.ConsoleMessage += OnBrowserConsoleMessage;
browser.StatusMessage += OnBrowserStatusMessage;
browser.TitleChanged += OnBrowserTitleChanged;
browser.AddressChanged += OnBrowserAddressChanged;
browser.FrameLoadEnd += browser_FrameLoadEnd;
browser.LifeSpanHandler = this;
browser.RequestHandler = this;
The declaration and the last two lines are the most important for this explanation. I implemented the IRequestHandler using the template found here:
https://github.com/cefsharp/CefSharp/blob/master/CefSharp.Example/RequestHandler.cs
I changed everything to what it recommends as default except for GetResourceResponseFilter which I implemented as follows:
IResponseFilter IRequestHandler.GetResourceResponseFilter(IWebBrowser browserControl, IBrowser browser, IFrame frame, IRequest request, IResponse response)
{
if (request.Url.EndsWith(".pdf"))
return this;
return null;
}
I then implemented IResponseFilter as follows:
FilterStatus IResponseFilter.Filter(Stream dataIn, out long dataInRead, Stream dataOut, out long dataOutWritten)
{
BinaryWriter sw;
if (dataIn == null)
{
dataInRead = 0;
dataOutWritten = 0;
return FilterStatus.Done;
}
dataInRead = dataIn.Length;
dataOutWritten = Math.Min(dataInRead, dataOut.Length);
byte[] buffer = new byte[dataOutWritten];
int bytesRead = dataIn.Read(buffer, 0, (int)dataOutWritten);
string s = System.Text.Encoding.UTF8.GetString(buffer);
if (s.StartsWith("%PDF"))
File.Delete(pdfFileName);
sw = new BinaryWriter(File.Open(pdfFileName, FileMode.Append));
sw.Write(buffer);
sw.Close();
dataOut.Write(buffer, 0, bytesRead);
return FilterStatus.Done;
}
bool IResponseFilter.InitFilter()
{
return true;
}
What I found is that the PDF is actually downloaded twice when it is loaded. In any case, there might be header information and what not at the beginning of the page. When I get a stream segment that begins with %PDF, I know it is the beginning of a PDF so I delete the file to discard any previous contents that might be there. Otherwise, I just keep appending each segment to the end of the file. Theoretically, the PDF file will be safe until you navigate to another PDF, but my recommendation is to do something with the file as soon as the page is loaded just to be safe.

Ignore user input while Uploading to FTP

I'm developing a windows store app, and I'm uploading a file to an FTP server with WebRequest, since it was the only work around I could find, with the limitations that I had.
When the application is uploading the video, which takes a few minutes, if the user taps the screen the app will crash. If no input is made, it will work fine.
When I was using Alex Pilotti's FTPS Client DLL, this didn't happen, but I couldn't get the certification for windows store using this DLL.
In my PC, this doesn't happen. It will wait until the video is uploaded and then execute the user input, but in the tablet it's a different story, maybe because it has less processing power/memory, it just crashes.
I was thinking: maybe there is a way to ignore all user input while the upload is happening.
I know it's not the best way, to take control from the user like that, but it would do the job and it would only be for a few minutes.
I've been googling, but I can't find a way to do this.
I'll leave my code below, just in case:
Public Async Function uploadFile(filename As String, file As StorageFile) As Task(Of Boolean)
Try
Dim ftpURL As String = "ftp://111.22.33.444"
Dim request As WebRequest = WebRequest.Create(ftpURL + "/" + filename)
request.Credentials = New NetworkCredential("user", "pass")
request.Method = "STOR"
Dim buffer As Byte() = Await ReadFileToBinary(filename, file)
Dim requestStream As Stream = Await request.GetRequestStreamAsync()
Await requestStream.WriteAsync(buffer, 0, buffer.Length)
Await requestStream.FlushAsync()
Return True
Catch ex As Exception
Return False
End Try
End Function
I fixed this issue with a very simple line of code:
Await Task.Run(Function() uploadFile(filename, file))
Worked for me.

WCF streaming SQLFileStream by Message Contract

In my WCF service, I try to load a File from MS SQL table which has a FileStream column and I try to pass it as a stream back
responseMsg.DocSqlFileStream = new MemoryStream();
try
{
using (FileStreamDBEntities dbEntity = new FileStreamDBEntities())
{
...
using (TransactionScope x = new TransactionScope())
{
string sqlCmdStr = "SELECT dcraDocFile.PathName() AS InternalPath, GET_FILESTREAM_TRANSACTION_CONTEXT() AS TransactionContext FROM dcraDocument WHERE dcraDocFileID={0}";
var docFileStreamInfo = dbEntity.Database.SqlQuery<DocFileStreamPath>(sqlCmdStr, new object[] { docEntity.dcraDocFileID.ToString() }).First();
SqlFileStream sqlFS = new SqlFileStream(docFileStreamInfo.InternalPath, docFileStreamInfo.TransactionContext, FileAccess.Read);
sqlFS.CopyTo(responseMsg.DocSqlFileStream);
if( responseMsg.DocSqlFileStream.Length > 0 )
responseMsg.DocSqlFileStream.Position = 0;
x.Complete();
}
}
...
I'm wondering whats the best way to pass the SQLFileStream back through a message contract back to take advantage of streaming. Currently I copied the SQLFilEStream to a memory stream because I got an error message in WCF trace which says: Type 'System.Data.SqlTypes.SqlFileStream' cannot be serialized.
In WebApi there is such thing as PushStreamContent it allows delegating all transaction stuff to async lambda, don't know if there is something similar in WCF, but the following approach may be helpful:
http://weblogs.asp.net/andresv/archive/2012/12/12/asynchronous-streaming-in-asp-net-webapi.aspx
You can't stream an SQLFileStream back to the client because it can only be read within the SQL transaction. I think your solution with the MemoryStream is a good way of dealing with the problem.
I had a similar problem and was worried about the large object heap when using a new Memory Stream every time. I came up with the idea of using a temporary file on the disk instead of a memory stream. We are using this solution in several project now and it works really well.
See here for the example code:
https://stackoverflow.com/a/11307324/173711

Stream text to client via handler ASP.NET

To get around twitters streaming API not having a crossdomain file to access it from client side( in this case Silverlight) I have made a Generic Handler file in a web project which basically downloads the stream from twitter and as it reads it, writes it to the client.
Here is the handler code:
context.Response.Buffer = false;
context.Response.ContentType = "text/plain";
WebRequest request = WebRequest.Create("http://stream.twitter.com/1/statuses/filter.json?locations=-180,-90,180,90");
request.Credentials = new NetworkCredential("username", "password");
StreamReader responseStream = new StreamReader(request.GetResponse().GetResponseStream(), Encoding.GetEncoding("utf-8"));
while (!responseStream.EndOfStream)
{
string line = "(~!-/" + responseStream.ReadLine() + "~!-/)";
context.Response.BinaryWrite((Encoding.UTF8.GetBytes(line)));}
And this does work, but the problem is that once the client disconnects the handler just carry's on downloading. So how do I tell if the client is still busy receiving the request and if not, end the while loop?
Also, my second problem is that on the client side doing a "ReadLine()" does nothing, presumably because it is counting the entire stream as one line so never gets the full response. To work around that I read it byte by byte and when it sees "(~!-/" around something it know that is one line. VERY hacky, I know.
Thanks!
Found the answer!
while (context.Response.IsClientConnected)
:)

Why am I getting System.FormatException: String was not recognized as a valid Boolean on a fraction of our customers machines?

Our c#.net software connects to an online app to deal with accounts and a shop. It does this using HttpWebRequest and HttpWebResponse.
An example of this interaction, and one area where the exception in the title has come from is:
var request = HttpWebRequest.Create(onlineApp + string.Format("isvalid.ashx?username={0}&password={1}", HttpUtility.UrlEncode(username), HttpUtility.UrlEncode(password))) as HttpWebRequest;
request.Method = "GET";
using (var response = request.GetResponse() as HttpWebResponse)
using (var ms = new MemoryStream())
{
var responseStream = response.GetResponseStream();
byte[] buffer = new byte[4096];
int read;
do
{
read = responseStream.Read(buffer, 0, buffer.Length);
ms.Write(buffer, 0, read);
} while (read > 0);
ms.Position = 0;
return Convert.ToBoolean(Encoding.ASCII.GetString(ms.ToArray()));
}
The online app will respond either 'true' or 'false'. In all our testing it gets one of these values, but for a couple of customers (out of hundreds) we get this exception System.FormatException: String was not recognized as a valid Boolean Which sounds like the response is being garbled by something. If we ask them to go to the online app in their web browser, they see the correct response. The clients are usually on school networks which can be fairly restrictive and often under proxy servers, but most cope fine once they've put the proxy details in or added a firewall exception. Is there something that could be messing up the response from the server, or is something wrong with our code?
Indeed, it's possible that the return result is somehow different.
Is there any particular reason you are doing the reasonably elaborate method of reading the repsonse there? Why not:
string data;
using(HttpWebResponse response = request.GetResponse() as HttpWebResponse){
StreamReader str = new StreamReader(response.GetResponseStream());
data = str.ReadToEnd();
str.Close();
}
string cleanResult = data.Trim().ToLower();
// log this
return Convert.ToBoolean(cleanResult);
First thing to note is I would definitely use something like:
bool myBool = false;
Boolean.TryParse(Encoding.ASCII.GetString(ms.ToArray()), myBool);
return myBool;
It's not some localisation issue is it? It's expecting the Swahili version of 'true', and getting confused. Are all the sites in one country, with the same language, etc?
I'd add logging, as suggested by others, and see what results you're seeing.
I'd also lean towards changing the code as silky suggested, though with a few further changes from me (code 'smell' issues, IMO); Use using around the stream reader, as well as the response.
Also, I don't think the use of as is appropriate in this instance. If the Response can't be cast to HttpWebResponse (which, admittedly is unlikely, but still) you'll get a NullRef exception on the response.GetResponseStream() bit which is both a vague error, and you've lost the original line number. Using (HttpWebResponse)request.GetResponse() will give you a more correct error, and the correct line number of the actual error.