I made a Azure cloud service which download files from a FTP server and then upload them to an Azure storage.
When I debug my service, everything works fine, the files are uploaded to the storage. Once I published the service, I received an error (not always, but 4 times of 5 try):
"System.Net.WebException: The operation has timed out."
The line number of the error correspond to the line
Dim response As FtpWebResponse = CType(request.GetResponse, FtpWebResponse)
Why does it work on local debug and not from cloud ?
For information, there is no proxy or what. It seems to work in a random way.
A part of my code :
'Request
Dim request As FtpWebRequest = CType(WebRequest.Create(Configuration.Address + suffixPath), FtpWebRequest)
request.Method = WebRequestMethods.Ftp.DownloadFile
'Login and pwd
request.Credentials = New NetworkCredential(Configuration.Login, Configuration.Password)
request.UsePassive = False
request.KeepAlive = True
request.Timeout = Integer.MaxValue
request.ReadWriteTimeout = Integer.MaxValue
Try
Dim response As FtpWebResponse = CType(request.GetResponse, FtpWebResponse)
Dim stream As Stream = response.GetResponseStream
...
Thx all
--- EDIT ----
I found this article :
http://feedback.azure.com/forums/217313-networking-dns-traffic-manager-vpn-vnet/suggestions/3346609-icmp-support-for-azure-websites-roles-cloud-serv
It's a suggestion to add ICMP traffic for Azure VMs, Services...
I tried to do a simple ping test from my Azure Cloud Service to my FTP Server, with a Timeout of 30 seconds, I received ... a timeout error immediatly.
Do you think guys that is a possible explanation of my problem ?
If it does, is there any means to download file from ftp by passing thru UDP or something else ?
I Finally found the solution to my problem.
The restriction for ICMP traffic inside a Azure VM or Service was the thing.
So, I simply used a WebClient object to download my files.
Dim webclient As New WebClient
webclient.Credentials = New NetworkCredential(login, password)
Dim bytesData As Byte() = webclient.DownloadData(addressOfFTPServer + fileNameToDownload)
Return bytesData
Hope this will help.
Related
We have problem with .NET application which runs fine on development PC (win10)
but when deployed to "Windows server 2016" production env it reports:
System.Net.WebException: The request was aborted: Could not create
SSL/TLS secure channel.
wireshark difference in Dev and Prod Env
On Dev (Win10) there is "Certificate,Client Key Exchange" after "Server Key Exchange" packet
On Prod (srv 2016) there is "TCP Spurious Retransmission" after "Server Key Exchange" as on screenshot
Tried with .NET 4.7.2 and 4.8 same error, how to proceed ?
It looks like there is problem on production server when PFX cert is used
Dim request As HttpWebRequest = CType(WebRequest.Create("https://obt2b1.service.com/cords/api/"), HttpWebRequest)
request.Method = WebRequestMethods.Http.Post
request.Accept = "application/xml"
request.ContentType = "application/xml"
'Use x509 PFX
request.ClientCertificates.Add(getCert())
request.KeepAlive = True
request.CachePolicy = New System.Net.Cache.RequestCachePolicy(System.Net.Cache.RequestCacheLevel.NoCacheNoStore)
request.Timeout = 1000000
request.AllowAutoRedirect = True
Dim dataStream As Byte() = Encoding.UTF8.GetBytes(xml_send)
request.ContentLength = dataStream.Length
ServicePointManager.Expect100Continue = true
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12
'Here the error "System.Net.WebException: The request was aborted: Could not create SSL/TLS secure channel." is reported
Dim _stream As Stream = request.GetRequestStream()
...
Private Function getCert() As X509Certificate2
Dim collection As New X509Certificate2Collection()
collection.Import("C:\pfx\test.pfx", "xxx", X509KeyStorageFlags.PersistKeySet Or X509KeyStorageFlags.UserKeySet Or X509KeyStorageFlags.MachineKeySet Or X509KeyStorageFlags.Exportable)
Return collection(0)
End Function
EDITED ... SecurityProtocol moved up same thing
ServicePointManager.Expect100Continue = true
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12
Dim request As HttpWebRequest = CType(WebRequest.Create("https://obt2b1.service.com/cords/api/"), HttpWebRequest)
request.Method = WebRequestMethods.Http.Post
request.Accept = "application/xml"
request.ContentType = "application/xml"
'Use x509 PFX
request.ClientCertificates.Add(getCert())
request.KeepAlive = True
request.CachePolicy = New System.Net.Cache.RequestCachePolicy(System.Net.Cache.RequestCacheLevel.NoCacheNoStore)
request.Timeout = 1000000
request.AllowAutoRedirect = True
Dim dataStream As Byte() = Encoding.UTF8.GetBytes(xml_send)
request.ContentLength = dataStream.Length
'GetRequestStream() gets the error "System.Net.WebException: The request was aborted: Could not create SSL/TLS secure channel." is reported
Dim _stream As Stream = request.GetRequestStream()
OK after an almost week of testing I found a solution.
Things that I tried:
windows server 2012R2 (working no problem)
windows server 2016 (not working)
windows server 2019 (not working)
I tried to enable ALL ciphers and ALL settings in IIS Cryto 3.2 - didn't help.
Problem was in worker process on the IIS, you need to enable "Load User Profile" to True
IIS -> Application Pools -> (choose pool) -> Advanced Settings ...
Change "Load User Profile" from false to true ...
After that I start IIS Crypto 3.2 set all setting to restrictive as it was before
I've made an experimental email and password system using post requests to my Wix site API and I've verified it works through curl and postman. When trying to use it in my VB.NET app however I get the error: "The underlying connection was closed: An unexpected error occurred on a receive." I already previously ran into a "Could not create SSL/TLS secure channel" error but that was resolved when I moved the .NET framework for the project to 4.7.1 and made sure SecurityProtocol was set to SystemDefault as Wix seems to use TLS 1.3. Now however I'm getting this error on receive that I have no idea how to resolve as everything in the request seems to look ok:
ServicePointManager.SecurityProtocol = SecurityProtocolType.SystemDefault
Dim req As HttpWebRequest = DirectCast(WebRequest.Create(New Uri("https://xxxxx.com/mysite/_functions/check?email=" & email & "&pass=" & LCase(passHash))), HttpWebRequest)
req.Method = "POST"
req.ContentType = "none"
req.ContentLength = 0
req.Host = "test"
req.KeepAlive = True
Dim result As HttpWebResponse = req.GetResponse()
The error is thrown when running 'req.GetReponse()'.
After looking at countless versions of sample code for .NET POST requests and questions about this error but everyone seems to suggest setting the security prococol type, which I have done and setting it as the system default is the only way I don't get an internal server error status. Another was setting KeepAlive to false, which I have also tried but didn't make any difference.
What could be the problem here? Here is what's included in the postman header if this is of help:
Ok so eventually I ended up changing my code to use GetResponseAsync() instead of GetResponse() (not that it made any difference to the problem) and set the protocol type back to Tls12 on .NET 4.8. In the end it seemed that having req.Host set was what was causing the issue with the response, Wix doesn't seem to like it. I also added some extra settings to the request but ultimately they make no difference to the response. Here is the final working code:
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12
Dim req As HttpWebRequest = WebRequest.CreateHttp("https://example.com/check?email=" & email & "&pass=" & LCase(passHash))
req.Method = "POST"
req.ContentType = "none"
req.ContentLength = 0
req.KeepAlive = True
req.CookieContainer = New CookieContainer()
req.UserAgent = "Test"
req.Accept = "ext/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"
req.Headers.Add(HttpRequestHeader.AcceptLanguage, "en-US;q=0.9,en;q=0.5")
req.Headers.Add(HttpRequestHeader.AcceptEncoding, "gzip, deflate;q=0.8")
req.Headers.Add(HttpRequestHeader.CacheControl, "no-cache")
Dim statusCode As HttpStatusCode = Nothing
Try
Using result As HttpWebResponse = CType(Await req.GetResponseAsync(), HttpWebResponse)
statusCode = result.StatusCode
End Using
If statusCode = HttpStatusCode.OK Then
MsgBox("Valid")
Else
Dim response As String = New StreamReader(req.GetResponse().GetResponseStream()).ReadToEnd()
MsgBox(response)
End If
Catch webEx As WebException
MsgBox(New StreamReader(webEx.Response.GetResponseStream()).ReadToEnd())
End Try
I'm a totally beginner with webrequest, so I have no idea about what cause the error I get.
I try to login on a form following the microsoft tutorial for webrequest, but when I want to get the server response, I have the following error :
"the remote server returned an error (404) not found"
So I know that the URL I use actually exist and then wonder which part of the code is bad. Maybe it's because I'm doing an HTTPS request unlike the tutorial and it changes something ?
Also, I'm a little confused by getting directly the answer from the server : shouldn't there be kind of a trigger to know when the server answered ?
Dim request = WebRequest.Create("https://ssl.vocabell.com/mytica2/login")
request.Credentials = CredentialCache.DefaultCredentials
request.Method = "POST"
Dim byteArray = Encoding.UTF8.GetBytes("_username=x&_password=x")
request.ContentType = "application/x-www-form-urlencoded"
request.ContentLength = byteArray.Length
Dim dataStream = request.GetRequestStream()
dataStream.Write(byteArray, 0, byteArray.Length)
dataStream.Close()
Dim reponse = request.GetResponse() 'ERROR
MsgBox(CType(reponse, HttpWebResponse).StatusDescription)
Using ds = reponse.GetResponseStream
Dim reader = New StreamReader(ds)
MsgBox(reader.ReadToEnd)
End Using
reponse.Close()
Thank you for your time, and if you have any relevant tutorial on the topic I would be glad to read it !
The page you've mentioned does exist and uses HTTPS, but if you look at the form tag within it, it's like this:
<form class="login-form form-horizontal" action="/mytica2/login_check" method="POST">
This means it doesn't post the form back to the same URL as the page, instead it sends it to the URL contained within that "action" attribute. If you're trying to use your code to simulate the submission of the login form then it looks like you need to send your POST request to https://ssl.vocabell.com/mytica2/login_check instead.
Regardless of whether I use WebClient or HttpWebRequest, loading this page times out. What am I doing wrong? It can't be https, since other https sites load just fine.
Below is my latest attempt, which adds all headers that I see in Firefox's inspector.
One interesting behavior is that I cannot monitor this with Fiddler, because everything works properly when Fiddler is running.
Using client As WebClient = New WebClient()
client.Headers(HttpRequestHeader.Accept) = "text/html, image/png, image/jpeg, image/gif, */*;q=0.1"
client.Headers(HttpRequestHeader.UserAgent) = "Mozilla/5.0 (Windows; U; Windows NT 6.1; de; rv:1.9.2.12) Gecko/20101026 Firefox/3.6.12"
client.Headers(HttpRequestHeader.AcceptLanguage) = "en-US;en;q=0.5"
client.Headers(HttpRequestHeader.AcceptEncoding) = "gzip, deflate, br"
client.Headers(HttpRequestHeader.Referer) = "http://www.torontohydro.com/sites/electricsystem/Pages/foryourhome.aspx"
client.Headers("DNT") = "1"
client.Headers(HttpRequestHeader.KeepAlive) = "keep-alive"
client.Headers(HttpRequestHeader.Upgrade) = "1"
client.Headers(HttpRequestHeader.CacheControl) = "max-age=0"
Dim x = New Uri("https://css.torontohydro.com/")
Dim data as string = client.DownloadString(x)
End Using
All of this is excess code. Boiling it down to just a couple of lines causes the same hang.
Using client as WebClient = New WebClient()
Dim data as string = client.DownloadString("https://css.torontohydro.com")
End Using
And this is the HttpWebRequest code, in a nutshell, which also hangs getting the response.
Dim getRequest As HttpWebRequest = CreateWebRequest("https://css.torontohydro.com/")
getRequest.CachePolicy = New Cache.RequestCachePolicy(Cache.RequestCacheLevel.BypassCache)
Using webResponse As HttpWebResponse = CType(getRequest.GetResponse(), HttpWebResponse)
'no need for any more code, since the above line is where things hang
So this ended up being due to the project still being in .NET 3.5. .NET was trying to load the site, being https, using SSL. Adding this line fixed the problem:
ServicePointManager.SecurityProtocol = 3072
I had to use 3072 since 3.5 does not contain a definition for SecurityProtocolType.Tls12.
I’m writing a http module to work as a reverse proxy, i.e. receives a request from a browser, sends it on to the target site, receives a response and sends that back to the browser.
Its all working fine, except for a problem with forwarding cookies from the browser request to the target site on a Post. All headers and form data are correct on the outgoing request, but no cookies are included.
I’ve run fiddler on both the request from the browser to IIS and the outgoing httpwebrequest and proven this to be the case. Running the module in debug shows that the cookies are found in the request from the browser and successfully placed in the cookiecontainer of the httpwebrequest, but they just don’t appear in the actual request sent out.
If I hack (in debug) the outgoing request method to a Get, then they go, but they don’t go for a Post.
I’ve also tracked the request/response from a browser direct to the target site using Fiddler, and the request seems identical in all three cases (browser to target, browser to my IIS module, IIS module to target), except that the IIS module to target omits the cookies.
Here’s the code (VB.Net, and tried in 2.0 and 4.5):
' set up the request to the target
Dim reqTarget As System.Net.HttpWebRequest
reqTarget = CType(System.Net.HttpWebRequest.Create(strTargetURL & strTargetPath & qstring), System.Net.HttpWebRequest)
' copy relevant info, cookies etc from the application request to the target request
CopyAppRequest(application.Context.Request, reqTarget)
' send the request and get the response
Dim rspTarget As System.Net.HttpWebResponse = CType(reqTarget.GetResponse(), System.Net.HttpWebResponse)
Private Sub CopyAppRequest(ByRef reqApp As System.Web.HttpRequest, ByRef reqTarget As System.Net.HttpWebRequest)
' copy over the headers
For Each key As String In reqApp.Headers.AllKeys
Select Case key
Case "Host", "Connection", "Content-Length", "Accept-Encoding", "Expect", "Authorization", "If-Modified-Since"
' not sure if we need to process these
Case "Connection"
reqTarget.Connection = reqApp.Headers(key)
Case "Content-Type"
reqTarget.ContentType = reqApp.Headers(key)
Case "Accept"
reqTarget.Accept = reqApp.Headers(key)
Case "Referer"
reqTarget.Referer = reqApp.Headers(key)
Case "User-Agent"
reqTarget.UserAgent = reqApp.Headers(key)
Case "Cookie"
' do nothing, cookies are handled below..
Case Else
reqTarget.Headers.Add(key, reqApp.Headers(key)
End Select
Next
reqTarget.Method = reqApp.HttpMethod
reqTarget.AllowAutoRedirect = False
If reqTarget.Method = "POST" Then
reqTarget.ContentLength = reqApp.ContentLength
Dim datastream() As Byte = System.Text.Encoding.UTF8.GetBytes(reqApp.Form.ToString)
reqTarget.ContentLength = datastream.Length
Dim requestwriter As System.IO.Stream = reqTarget.GetRequestStream
requestwriter.Write(datastream, 0, datastream.Length)
requestwriter.Close()
requestwriter.Dispose()
End If
Dim CookieJar As New System.Net.CookieContainer
reqTarget.CookieContainer = CookieJar
For Each key As String In reqApp.Cookies.AllKeys
Dim tgtCookie As New System.Net.Cookie
With tgtCookie
.Name = reqApp.Cookies.Item(key).Name
.Value = reqApp.Cookies.Item(key).Value
.Domain = ".domain.com"
.Path = "/"
.Expires = DateAdd(DateInterval.Month, 1, System.DateTime.Now)
.HttpOnly = True
End With
CookieJar.Add(tgtCookie)
Next
End Sub
Note: the domain I’m trying to reach is in the form abc.domain.com (i.e. it’s a subdomain, and no www), the reason I’ve tried the .domain.com form is that is the form used in the cookies that are received in the response. I’ve also tried other combinations such as abc.domain.com, .abc.domain.com, etc. Also I’ve tried creating a Uri object and using that method to add the cookie into the cookiecontainer.
I’ve tried everything I can think of and can find on forums…. Anyone got any suggestions? I suspect I’ve missed something obvious!
Of course, any other comments on how the code above can be improved will be appreciated.
Thanks.
Ok, I found the issue...
I was using Fiddler to see what was going on with http, and Fiddler was correct. However, when I used Wireshark, I found that the http request was being sent earlier than I thought, and only on the Post.
It turned out that the requestwriter.write line caused the http request to be sent, not the GetResponse (as is the case with Get). So, anything I changed in the httpwebrequest after the requestwriter.write didn't get sent.
The fix - I just moved all the header and cookie set up above the requestwriter.write and it all worked.
How frustrating, but at least its fixed now :)
If anyone has any feedback on whether I've got something wrong that's causing this to happen, please let me know.