ProtocolError while calling HttpWebRequest.GetResponse() - httpwebrequest

Question

I have a page containing links to some files.
I basically need to access the source of the page for parsing it then and obtaining all the hyperlinks to the files.
My code is something like this (some piece of code I've found in many places on the net ..):
"private static byte[] ReadImageFromUrl(string url)
{
var myReq = (HttpWebRequest)WebRequest.Create(url);
myReq.Timeout = 10000;
WebResponse myResp = myReq.GetResponse();
Stream stream = myResp.GetResponseStream();
List<byte> bytesList = new List<byte>();
using (var br = new BinaryReader(stream))
{
try
{
while (true)
{
var b = br.ReadByte();
bytesList.Add(b);
}
}
catch (Exception)
{}
br.Close();
}
myResp.Close();
return bytesList.ToArray();
}"
Now the problem is I get "System.Net.WebException: The remote server returned an error: (500) Internal Server Error." when calling "myReq.GetResponse()" - examining the error I see that the status is 'ProtocolError'.
The response property of the WebException object contains some server error ..(although when opening it from the browser it opens correctly) ...also when I call this function with the url of one of my files I get the same ProtocolError status, but the 404 error ...
Please give any hint how could I solve it... or any other possibility of accomplishing this task.
Thanks !

Answer 1

My new code after using Fiddler is:
private static byte[] ReadFileFromUrl(string url)
{
var myReq = (HttpWebRequest)WebRequest.Create(url);
myReq.Accept = const_AcceptHeader;
myReq.Headers.Set(const_AcceptLanguageHeaderName, const_AcceptLanguageHeader);
myReq.UserAgent = const_AcceptUserAgentHeader;
myReq.CookieContainer = new CookieContainer();
myReq.KeepAlive = true;
myReq.Timeout = Int32.Parse(ConfigSettings.RequestPageTimeout) * 1000;
WebResponse myResp = null;
List<byte> bytesList = null;
myResp = myReq.GetResponse();
Stream stream = myResp.GetResponseStream();
bytesList = new List<byte>();
using (var br = new BinaryReader(stream))
{
try
{
while (true)
{
var b = br.ReadByte();
bytesList.Add(b);
}
}
catch (Exception ex)
{
throw;
}
br.Close();
}
return bytesList.ToArray();
}
All variables that start with const_ are taken from Fiddler.

Answer 2

Well, I solved that using Fiddler ... I passed to my request object the headers as I have seen them in Fiddler ...& it worked, no error