I have a tool that loads subtitle from subscene.com, I wrote the code in vb.net, it used to work until 21th of Match 2022.
The strange thing is when I open the website using google chrome, although I delete all cookies, site data, storage and etc data, there is no challenge in browser, I don't see any redirection in developer toolbar/network, but when I use WebClient it returns cloudflare challenge page in webexception response data.
I simulated every page header that chrome sends, using same IP and user agent and still no luck.
I have been struggling this for few days but I couldn't figure it out how cloudflare distinguishes my request that chrome request? can anybody help me?
Below is the source of the asp.net generic handler HttpGet.ashx
The usage is [Path]/HttpGet.ashx?url=http://subscene.com
It returns html data for any website given.
<%# WebHandler Language="VB" Class="GetHtml" %>
Imports System.IO
Imports System.Net
Public Class GetHtml : Implements IHttpHandler
Public Sub ProcessRequest(ByVal context As HttpContext) Implements IHttpHandler.ProcessRequest
Dim url As New Uri(HttpUtility.UrlDecode(context.Request.QueryString("url")))
dim wc = new WebClient()
wc.Headers.Add("Accept-Language", "en-US")
wc.Headers.Add("Accept", "*/*")
wc.Headers.Add("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36")
Try
dim html=WC.DownloadString(url.AbsoluteUri)
context.Response.ContentType=wc.ResponseHeaders("Content-Type")
context.Response.Write(html)
Catch ex As WebException
Dim st As Stream = ex.Response.GetResponseStream()
Dim sr As New StreamReader(st)
Dim html = sr.ReadToEnd
st.Close()
context.Response.ContentType=ex.Response.ContentType
context.Response.Write(html)
Catch ex As Exception
Throw New Exception(ex.Message, ex)
End Try
End Sub
Public ReadOnly Property IsReusable() As Boolean Implements IHttpHandler.IsReusable
Get
Return False
End Get
End Property
End Class
Related
I have an app which will download publicly available data as a string from a website.
The below code on a WinForm app on Windows 10 works perfectly, but it doesn't work at all on Windows Server 2019.
I have tried enabling the "Allow App Through Firewall" option & have also tried allowing the program in outbound rules in the firewall, but nothing is working.
What is the issue?
Private Async Function Run_WebRequest(URL As String) As Task(Of String)
Try
Dim client As WebClient = New WebClient()
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12
client.Headers.Add("scheme", "https")
client.Headers.Add("accept-language", "en-US, en;q=0.9, ml;q=0.8")
Dim Response = client.DownloadString(URL)
Return Response
Catch ex As Exception
Return ex.ToString
End Try
End Function
I missed to add that this is a Virtual Private Server OS Windows Server 2019 from Contabo hosting services
Please see below screenshot of the app which is a standalone exe file with three buttons having different Webclients for testing, none of them works on
I'm using the below code to try and access an API using RestSharp in VB.Net:
Dim myClient As New RestClient(*https api website url*)
Dim myRequest As New RestRequest(Method.POST)
myRequest.AddHeader("Content-Type", "application/json")
myRequest.AddCookie("JSESSIONID", "568E16300DB90F7CEC5A7882C66D3684")
myRequest.AddCookie("sessionExpiry", "3600000") ' 1 Hour
myRequest.AddParameter("undefined", "{\""username\"":\""*myUsername*\"",\""password\"":\""*myPassword*\"",\""clientInformation\"":{\""userAgent\"":\""Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Safari/537.36\"",\""height\"":1440,\""width\"":2560,\""os\"":\""Win32\""},\""application\"":\""p4app\""}", ParameterType.RequestBody)
Dim myResponse As IRestResponse
myResponse = myClient.Execute(myRequest)
MsgBox(myResponse.Content)
it keeps giving me the error '"message":"Login Failed - Invalid login request"'
I've been told I need to ignore the SSL certificate but I can't seem to find a way to do that using RestSharp. Any ideas?
I don't know what RestSharp is using underneath, but it might work like this:
Imports:
Imports System.Net
Imports System.Net.Security
Imports System.Security.Cryptography.X509Certificates
Register a certificate validator:
Dim myHandler As RemoteCertificateValidationCallback =
Function(sender As Object, certificate As X509Certificate, chain As X509Chain, sslPolicyErrors As SslPolicyErrors) As Boolean
Return True
End Function
ServicePointManager.ServerCertificateValidationCallback = myHandler
I am trying to setup an "external" web api that will be able to receive in large http posts (+1GB) and forward the stream to another "internal" web api that writes the request contents to a file. I have model my implementation based on examples of using a custom WebHostBufferPolicySelector and using the UseBufferedInputStream method in the controller method. It works as expected when using IIS Express, no significate increase in memory footprint, but as soon as my code is deployed to IIS, the memory footprint is substantial and results in OOM.
I have put tracing statements in my controller methods and in my WebHostBufferPolicySelector.UseBufferedInputStream, and have verified that UseBufferedInputStream always is returning false and that my controller methods are getting hit. The only difference that I noticed is that when I debug, the time stamps between UseBufferedInputStream and my controller method are very close. Where hosted on IIS, the time stamps are very far apart, suggesting that something in between when UseBufferedInputStream is called and my controller method is called is buffering up the request entirely.
I am looking for some tips on to find out what that is causing the request to get buffered and how for it not to buffer and using streaming all the way.
Client is coming at the external web api with a content type of application/octet-stream with Transfer Encoding of Chucked.
Used to build out implementation
https://forums.asp.net/t/2018289.aspx?Web+API2+WebHostBufferPolicySelector+UseBufferedInputStream+override
https://www.strathweb.com/2012/09/dealing-with-large-files-in-asp-net-web-api/
Proxy Web Api Controller Method
<HttpPost, Route("postLargeFile")>
Protected Overridable Async Function PostLargeFile() As Threading.Tasks.Task(Of IHttpActionResult)
Configuration.Services.GetTraceWriter.Info(Request, $"{Me.GetType.Namespace}.{NameOf(MyProxyController)}", "Started {0}", NameOf(MyProxyController.PostLargeFile))
Dim internalHttpClient As HttpClient
Dim fowardingContent As StreamContent = Nothing
Dim fowardingMessage As HttpRequestMessage = Nothing
Dim fowardingResponse As HttpResponseMessage = Nothing
Dim externalResponse As HttpResponseMessage = Nothing
Try
internalHttpClient = New HttpClient()
internalHttpClient.BaseAddress = "https://myinternalService.com"
fowardingMessage = New HttpRequestMessage(HttpMethod.Post, "https://myinternalService.com/saveLargeFile")
fowardingContent = New StreamContent(HttpContext.Current.Request.GetBufferlessInputStream(True))
CopyContentHeaders(Request.Content, fowardingContent)
fowardingMessage.Headers.TransferEncodingChunked = True
fowardingMessage.Content = fowardingContent
fowardingResponse = Await internalHttpClient.SendAsync(fowardingMessage, HttpCompletionOption.ResponseHeadersRead)
externalResponse = New HttpResponseMessage(fowardingResponse.StatusCode)
externalResponse.Content = New StreamContent(Await fowardingResponse.Content.ReadAsStreamAsync)
CopyContentHeaders(fowardingResponse.Content, externalResponse.Content)
Return New Results.ResponseMessageResult(externalResponse)
Catch ex As Exception
Return InternalServerError(ex)
Finally
Configuration.Services.GetTraceWriter.Info(Request, $"{Me.GetType.Namespace}.{NameOf(MyProxyController)}", "Finished {0}", NameOf(MyProxyController.PostLargeFile))
End Try
End Function
Internal Web Api Controller Method
<HttpPost, Route("saveLargeFile")>
Protected Overridable Async Function SaveLargeFile() As Threading.Tasks.Task(Of IHttpActionResult)
Configuration.Services.GetTraceWriter.Info(Request, $"{Me.GetType.Namespace}.{NameOf(MyInternalController)}", "Started {0}", NameOf(MyInternalController.PostLargeFile))
Dim bufferlessStream As IO.Stream
Dim fowardingContent As StreamContent = Nothing
Try
bufferlessStream = HttpContext.Current.Request.GetBufferlessInputStream()
Using fileStream As IO.FileStream = IO.File.Create("MyFile.txt")
bufferlessStream.CopyTo(fileStream)
fileStream.Flush()
End Using
Return New Results.StatusCodeResult(Net.HttpStatusCode.Created, Me)
Catch ex As Exception
Return InternalServerError(ex)
Finally
Configuration.Services.GetTraceWriter.Info(Request, $"{Me.GetType.Namespace}.{NameOf(MyInternalController)}", "Finished {0}", NameOf(MyInternalController.PostLargeFile))
End Try
End Function
Policy Selector Configuration
Public Class MyBufferPolicySelector
Inherits Http.WebHost.WebHostBufferPolicySelector
Public Property Tracer As ITraceWriter
Public Overrides Function UseBufferedInputStream(hostContext As Object) As Boolean
UseBufferedInputStream = False
Tracer?.Info(Nothing, $"{Me.GetType.Namespace}.{NameOf(MyBufferPolicySelector)}", "{0} UseBufferedInputStream={1}", HttpContext.Current?.Request?.Url?.AbsoluteUri, UseBufferedInputStream)
Return UseBufferedInputStream
End Function
End Class
WebApiConfig for both Internal and External Web APIs
Public Module WebApiConfig
Public Sub Register(ByVal config As HttpConfiguration)
Dim tracer As SystemDiagnosticsTraceWriter
' Web API configuration and services
' Web API routes
config.MapHttpAttributeRoutes()
tracer = config.EnableSystemDiagnosticsTracing
tracer.IsVerbose = True
tracer.MinimumLevel = Tracing.TraceLevel.Debug
GlobalConfiguration.Configuration.Services.Replace(GetType(IHostBufferPolicySelector), New MyBufferPolicySelector() With {.Tracer = tracer})
End Sub
End Module
I was able to figure out what was causing the the buffering in IIS. The below link lead me to the uploadReadAheadSize setting in IIS. This was maxed out. So this would cause IIS to fully read in/buffer in the request before passing it into the module where the web api pipeline exists (web api controllers). After setting it to the default, I saw my large file posts not get buffered, the app pool memory footprint remained low, no more out of memory exceptions, and a large performance boost. Great!
But now I have the same issue as described in the below link. When SSL is required, set in IIS, which it is required in our non development environments, the uploadReadAheadSize needs to be increased so the ssl can work in the ssl module I guess. It might have to do with some SSL renegotiation.
Can anybody describe a way to prevent the buffering in SSL to keep the memory footprint low and prevent out of memory exceptions for large http posts?
Large file upload when using ssl and client certificates (uploadReadAheadSize) but dont want all data to be readahead
In a Web API 2 web service I currently have a custom authentication handler set up for the default route in WebApiConfig.vb, the default route looks like this:
config.Routes.MapHttpRoute(
name:="DefaultApi",
routeTemplate:="api/{controller}/{id}",
defaults:=New With {.id = RouteParameter.Optional},
constraints:=Nothing,
handler:=New AuthenticationHandler() With {.InnerHandler = New Dispatcher.HttpControllerDispatcher(config)}
)
And the AuthenticationHandler class looks like this:
Imports System.Net
Imports System.Net.Http
Imports System.Threading.Tasks
Imports System.Runtime.Caching
Imports System.Text
Imports System.Security.Cryptography
Public Class AuthenticationHandler
Inherits DelegatingHandler
Protected Overrides Function SendAsync(request As HttpRequestMessage, cancellationToken As Threading.CancellationToken) As Threading.Tasks.Task(Of HttpResponseMessage)
Dim response As New HttpResponseMessage
Dim lblnIsAuthenticated As Boolean = False
Dim tsc As New TaskCompletionSource(Of HttpResponseMessage)
'// Do the authentication work here...
if not lblnIsAuthenticated then
'// Set the response
response.StatusCode = HttpStatusCode.BadRequest
response.Headers.Add("Status-Message", "Authentication failed.")
tsc.SetResult(response)
Return tsc.Task
end if
Return MyBase.SendAsync(request, cancellationToken)
End Function
When I upgraded this service to Web API 2 I wanted to take advantage of http attribute routing, so I added config.MapHttpAttributeRoutes() to WebApiConfig.vb and set up some of the controllers to use the Route and RoutePrefix attributes. The problem is that this routing method doesn't take advantage of the custom authentication handler. Is there any way to implement this same authentication handler for the http attribute routing? I've searched around for a solution to this but don't see how to make this work. Or do I need to just do standard routes?
UPDATE --
I also have some other routes in WebApiConfig.vb that have a different authentication handler on them so I don't think I can apply one handler globally.
You should be able to add your handler to GlobalConfiguration.Configuration.MessageHandlers; unless I'm mistaken the handlers added there are always invoked for all the requests. Unless you're trying to do something different?
I wrote simple application on VS2010 that send httpwebrequest and without any configurations fiddler is captures this request. But after, I installed VS2012 and run fiddler, and when i send request i have exception "Operation timed out" and request is no captured. When i close fiddler all requests are sends.
I delete VS2012 and .net framework 4.5. After that request are sends and fiddler capturing them.
Why fiddler dont't capture traffic when .net4.5 installed?
Did you by any chance try to set the Host property of the HttpWebRequest?
This may be the cause of your problem.
I have also .NET 4.5 installed and experience the same situation.
I get the same error when fiddler is running and is acting as a proxy. The error is:
System.Net.WebException: The operation has timed out at
System.Net.HttpWebRequest.GetResponse()
Here is a trivial sample that reproduces the problem:
using System;
using System.IO;
using System.Net;
namespace WebRequestTest
{
class Program
{
static void Main(string[] args)
{
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://www.microsoft.com");
request.Host = "www.microsoft.com";//If I comment this line, capturing with fiddler works OK.
request.Method = "GET";
request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:18.0) Gecko/20100101 Firefox/18.0";
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
using (Stream stream = response.GetResponseStream())
using (StreamReader sr = new StreamReader(stream))
{
string content = sr.ReadToEnd();
Console.WriteLine(content);
}
}
}
}
In my case I just had to comment the request.Host="www.microsoft.com" line and everything worked OK.
I suspect same behavior will occur when using an HTTP proxy other than fiddler, but I have not tested it though.