I need to understand how exactly works the bot-detection works on axs.com as I'm accessing many times this website and after a short while the login does not work any more and website is basically blocked.
By analyzing the network request and console I can see the following after clicking the login button:
Access to XMLHttpRequest at 'https://api.axs.com/proxy/v3/flash/migrate/authenticate?access_token=4f2be33d835e7197e245c54ff00e5fb4®ion=1&locale=en-US' from origin 'https://www.axs.com' has been blocked by CORS policy: Response to preflight request doesn't pass access control check: No 'Access-Control-Allow-Origin' header is present on the requested resource.
WHAT I HAVE TRIED SO FAR:
Using multiple VPNs to change IP address (Failed)
Using an EC2 instance (failed)
Using Extension called "Trace" to completely delete all my trackable data and delete cookies (Failed)
Incognito window (Failed)
Proxy Server Paid and Free (Failed)
What else can I try?
I finally found the solution, it was needed only a better quality proxy. My steps:
At the beginning we used rotating proxy with brightdata from many different countries, but no luck with it.
Then we tried different proxy providers but always rotating, residential and mobile were tested. Again no luck.
We purchased US proxy from https://www.proxyrack.com/ and finally it worked. I guess the proxy quality is of vital importance. AXS security must have a pretty long blacklist for IP addresses.
Related
Lets say I have my website named SiteA.com running on an Apache web server. I have defined the ff. below on my httpd.conf file:
Header set Access-Control-Allow-Origin "CustomBank.com"
Questions:
Does this mean only CustomBank.com can access my site (SiteA.com) directly? or does it mean only my site (SiteA.com) can access the CustomBank.com domain directly? I am confused if this setting is for inbound or outbound.
In reality I don't have any CORS requirement needed for my site, so I didn't implement the setting mentioned above, the one below shows up in my response header.
Access-Control-Allow-Origin: *
Penetration Testing team said this setting is overly permissive. Do I just need to remove it? if not what should I do?
It means javascript loaded from CustomBank.com can make requests to your site (the site whose configuration has changed) via XMLHTPRequest in the background.
Since XMLHTTPRequest will send a users existing session cookie with your site, malicious scripts could do all kinds of nefarious/misleading things on behalf of your user. That's why * is not normally a suitable fix.
The restrictions apply to other script-like invocations that are more esoteric that you can read about in the specs.
We use a custom HTTP module in IIS as a reverse proxy for web applications. Generally this works well and has done for some time, but we've come across an issue with Windows Authentication (WA). We're using IE 11, IIS 10 and Server 2016.
When accessing the target site directly, WA works fine - we get a browser login dialog when the initial HTML page is requested and the subsequent requests (CSS, JS, etc) go through fine.
When accessing via our proxy, the same (correct behaviour) happens for the initial html page, the first CSS/JS request authenticates ok too, but the subsequent ones cause a browser login to popup.
What seems to happen on the 'bad' requests (i,.e. those that cause the login dialog) is:
1) Browser decides it needs to authenticate, so sends an Authorization header (Negotiate, with an NTLM token)
2) Server responds (401) with a WWW-Authenticate: Negotiate response with a full NTLM token
3) Browser re-requests with an Authorization header (Negotiate, with a full NTLM token)
4) Server responds (401) with a WWW-Authenticate: Negotiate (with no token), which causes the browser to show the login dialog
5) With login credentials entered, Browser sends the same request as in (1) - identical NTLM token, server responds as in (2), Browser re-requests as in (3), but this time it works!
We've set up a test web site with one html page, requesting 3 JS and 2 CSS files to replicate this. On our test server we've got two sites, one using our reverse proxy and one using ARR. The ARR site works fine. Also, since step (5) above works, we believe that the proxy pass-through is fundamentally working, i.e. NTLM tokens are not being messed up by dodgy encoding, etc.
One thing that does work, is that if we use Fiddler and put breakpoints on each request, we're able to hold back on the 5 sub-requests (JS & CSS files), letting one go through at a time. If we let each sequence (i.e. NTLM token exchange for each URL/file, through to the 200 response), then it works. This made us think that there is some inter-leaving effect (e.g. shared memory corruption) in our proxy, this is still a possibility.
So, we put code at the start of BeginRequest and end of EndRequest with a Synclock and a shared var to store the Path (AppRelativeCurrentExecutionFilePath). This was for our code to 'Single Thread' each of these request/exchanges. This does what we expected, i.e. only allowing one auth exchange to happen and resulting in a 200 before allowing the next. However, we still have the same problem of the server rejecting the first exchange. So, does this indicate something happening in/before BeginRequest, where if we hold the requests back in Fiddler then they work, but not if we do it in our http module?
Or is there some sort of timing issue where the manual breakpoints in Fiddler also mean we’re doing it at ‘human’ speed and therefore allowing things to work better?
One difference we can see is the ‘Connection: Keep-Alive’. That header is in the request from the browser to our proxy site, but not passed from our proxy to the base site, yet the ARR site does pass that through... It’s all using HTTP 1.1. and so we can't find a way to set Keep-Alive on our outgoing request - could this be it?
Regarding 'things to try', we think we've eliminated things like having the site in the Intranet Zone for IE by having the ARR site work ok, and having the same IE settings for that site. Clearly, something is not right, so we could have missed something here!
In short, we've been working on this for days, and have tried most of what we can find on SO and elsewhere, but can't figure out what the heck is going on.
Any suggestions - let me know if you want any further info. All help will be very gratefully received!
Our website uses cloudflare as it's CDN to handle loads.
One of our apps requests the URL http://www.codenameone.com/files/cn1libs/CN1JSON.cn1lib with a get request. This works fine for every machine/location we tested but we have user complaints that they are getting an HTTP 411: "Length Required" response.
Since this is a GET request content-length doesn't seem like a header we would need to send...
Our server logs don't show any 411 response so my only conclusion is that this is a failure on the cloudflare side. However, since we can't reproduce this and the cloudflare aspect is a black box I don't have much to go on in terms of debugging.
I tried contacting cloudflare support but effectively got the usual "run around" asking me to send traces from a users machine on the other side of the world which is not something I can realistically do.
After a long session with cloudflare support it seems that unless you are an enterprise subscriber log files for access just don't exist. So effectively their support sees cloudflare as a black box just like we do.
Since the problem clearly isn't in our servers my educated guess is that this is a bug in cloudflare related to some odd edge case.
If someone has a better answer than this I'll gladly accept it.
I created an instance group through an instance template, and aligned this instance group to a backend service which is used by a http load balancer.
Now when I open a url to an instance vm from the instance group I created, I can do GET POST and DELETE requests and all of the requests are fast, and everything works as expected.
When I open up the url to the static IP for the load balancer. I can do GET and POST requests, but DELETE requests throw a 400 BAD REQUEST with a response page saying:
That’s an error.
Your client has issued a malformed or illegal request. That’s all we
know.
Other load balancer issues:
The site is quite slow through the load balancer. Perhaps
there is a setting I'm missing, I'm pretty sure I set everything to
us-central-1b.
Sometimes the site doesn't even show up. It will work for http, but then
it won't work for https and visa versa. The load balancer has very strange
behaviour.
My VM api access is set to This instance has full API access to all Google Cloud services
I'm using Django as my api layer, I turned on debugging on this host and saw that the DELETE requests weren't even coming through when making requests through the loadbalancer static ip. Is there a firewall setting I'm missing?
Please help me make this fast again and allow the DELETE requests to happen.
Thanks!
Are you sending anything in the body of the request?
Google load balancer will respond with 400 BAD REQUEST if you try to send anything in the body. Easy way to check if this is the problem is fire up Chrome Developer tools and check the Request Payload section is empty/doesn't exist.
The HTTP spec doesn't explicitly say wether you can pass anything in the body so this isn't wrong, just undefined.
Is the load balancer slow for all requests or just pages with lots of elements on?
I'm keeping the backend API as a separate project from the frontend HTML5 app consuming it. I'm using Yeoman for the frontend development. Yeoman runs on localhost:3501 and the backend on localhost:3000. When I make the API request from the browser (using AngularJS's $http), I hit the same origin policy:
XMLHttpRequest cannot load http://localhost:3000/venues. Origin http://localhost:3501 is not allowed by Access-Control-Allow-Origin.
AFAIK, same origin policy should kick in only when making request across different domains. Why is it whining when we do a request from localhost to localhost (albeit to different port)?
How can I make this work and will this cause problems in production?
The ports also count for cross domain requests, therefore http://localhost:3000 and http://localhost:3501 are 2 different domains (from browser's point of view).
If you need both applications (client and backend) to run on different ports, consider using http://enable-cors.org/
According to W3C, "An origin is defined by the scheme, host, and port of a URL", so the different port is causing your problems.
Two possible solutions:
CORS (Cross Origin Resource Sharing)
Using JSONP requests
Both would require changes to your backend (I'm not familiar enough wo. CORS would probably mean the least changes to your frontend (I think AngularJS supports it out-of-the-box).