Since adding HTTP security headers, Ahrefs.com produces JS 404s - http-headers

We're using WP Engine for our website host. I added some Web Rules to produce the following HTTP headers:
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Referrer-Policy: origin-when-cross-origin
Feature-Policy: geolocation 'self'
X-Frame-Options: SAMEORIGIN
Content-Security-Policy: upgrade-insecure-requests
Since making the change above, the next Ahrefs.com Site Audit produced a lot of 404s for some Beaver Builder JS files.
If you load the webpage where that 404 is detected by Ahrefs, then there are no broken resources.
It is only the Ahrefs user agents which are generating the 404 for some Beaver Builder JS files.
Could the HTTP headers added above be producing these 404s by the Ahrefs user agent?
WP Engine say the Ahrefs user agents are not blocked.
Help appreciated.
Update: in the Ahrefs crawl settings, turning on "Execute Javascript " resolved this issue for us.

Some time ago Arefs announced rendering web pages and executing JavaScript for its Crawler and for Site Audit tool.
To do this, Ahrefs had to partially implement browser rendering. This was not a full-fledged browser implemented, but some part of its "ready to use" code was taken. This renderer, inherited partial support of HTTP headers, which was built into this "ready-to-use" code.
But the main task was to search for links inserted via javascript, and not full support for HTTP headers.
Therefore, the implementation used incorrectly executes some headers, which one it can only be found out by experimenting - disable them one by one and wait for the results.
But these are hardly headers Referrer-Policy, Feature-Policy and X-Frame-Options - they cannot result in a 404 Not Found error.
Most likely, these are just an Ahrefs renderer errors of loading external scripts, they are unlikely to have an impact on the analysis capabilities of Ahrefs or SE ranking.

Related

My new release not getting browsed in browser post update

I have a website hosted in Apache. its a spring framework application uses angular for front end.
This is being accessed through akamai technology. I have updated the application hosted in the origin server.
When I tried to access from browser it gives a blank page. After doing an 'CTRL + F5' its gives me proper website.
I am not able to sort this out why such behavior in the application. Can any expert in this help me out.
Thanks In advance
Bala
Your page might be cached in your browser. I'd recommend that you could check "cache-control" HTTP response header, e.g.
cache-control: max-age=0, no-cache
It can be controlled by either Akamai or Apache.

How to enable safari browser caching for urls which are 302 redirects

I have a single-page application which is dependent on a javascript bundle to work. For fetching this bundle's CDN (cloudfront) url, I'm making a call to an AWS API Gateway endpoint which returns a HTTP 302 response having the Location header parameter as the CDN url. Now this CDN Url responds with cache-control headers having a sufficiently large max-age value. All the other browsers like Chrome, Firefox seem to honor this and cache the CDN Url response for further requests. But Safari isn't doing so (Version - 12). However, it does cache the response when I'm making the request to the CDN Url directly. Do I need to add some more headers or some additional metadata in the 302 response to make it work for safari?
I tried fiddling with the cache-control parameters like adding 'immutable' but nothing worked. I googled quite a lot about this issue but nothing concrete turned up.
I expected Safari to work with just the max-age parameter present in CDN's response, but it never caches it.

Referrer and origin preflight request headers in Safari are not changing when user navigates

I have two web pages hosted on a.example.com and b.example. Each web page is including a script with a <script> tag, hosted on another domain and served with correct CORS headers.
At a certain point, user navigates from a.example.com to b.example.com.
Safari has here a strange behavior: the referrer and origin headers in preflight request are filled with a.example.com, making the server sending a bad value in Access-Control-Allow-Origin (and so the script can't be executed).
Is there a way to force Safari browser to send correct origin header in that kind of scenario ?
Does the cache policy for the script include Vary: Origin?
Respectively is there actually a second request after navigating to b.example.com?
If not, there is a chance that Safari is actually serving the script from cache - despite the Access-Control-Allow-Origin policy forbidding it to access the resource. Which is a conforming behavior, if the cache policy isn't configured correctly.
It seemed to be indeed a Safari bug.
The issue is not reproductible on Safari 10.0. It repro only on Safari 9.1.1 / 9.1.3.

BlazeDS data push over SSL

I have an application that uses the data push technology of blazeDS to send data to a Flex Client event 5 seconds. The application works fine when I run it via HTTP with or without a proxy. When I run it via https the data push doesn't work anymore. I get the following error
rootCause [IOErrorEvent type="ioError" bubbles=false cancelable=false eventPhase=2
text="Error #2032: Stream Error.
URL: https://localhost/admin/messagebroker/streamingamfsecure?command=open&version=1
Has anyone successfully got streaming to work over SSL?
Thanks,
Pratima
Questions to ask yourself (and post here)
Is the request showing up in your access logs?
Does Tomcat/whatever server up normal HTML pages via HTTPS?
What do the response headers look like? Does clearing your cache change anything?
What browser are you using?
Can you set explicate caching headers?
Try one of these:
Cache-Control: no-store
Cache-Control: no-store, must-revalidate
Cache-Control: no-store,max-age=0,must-revalidate
Cache-Control: max-age=0,must-revalidate
Cache-Control: must-revalidate
2032 is a bit of a vague error from the framework.
However, things to check (in addition to Stu's list)
Can you hit the https:// page in a browser directly?
I notice in your example that you haven't specified the port number for SSL. Unless you've gone to the trouble of setting up some Apache SSL redirects, chances are this is a mistake.
If you paste the URL into a browser, you should be able to hit it, and get an empty response. Anything else, and you've got a problem (often one that doesn't relate to BlazeDS.)
Is your certificate valid?
If you're using a Self signed cert (as is common in development), does your browser have a security exception defined? Various browsers will block attempts to hit invalid certs in different ways, but no self-resepcting browser would allow this call through until an exception has been set up.
Is your channel defined correctly?
When switching from http:// to https://, you need to update your Channel class on the flex client to SecureAMFChannel and the endpoint class in your services-config.xml to SecureAMFEndpoint.
Broadly speaking, https with BlazeDS (either push, or RPC) works just fine, assuming you configure it properly.

How do I figure out which parts of a web page are encrypted and which aren't?

I'm working on a webserver that I didn't totally set up and I'm trying to figure out which parts of a web page are being sent encrypted and which aren't. Firefox tells me that parts of the page are encrypted, but I want to know what, specifically, is encrypted.
The problem is not always bad links in your page.
If you link to iresources at an external site using https://, and then the external site does its own HTTP redirect to non-SSL pages, that will break the SSL lock on your page.
BUT, when you viewing the source or the information in the media tab, you will not see any http://, becuase your page is properly using only https:// links.
As suggested above, the firebug Net tab will show this and any other problems. Follow these steps:
Install Firebug add-on into firefox if you don't already have it, and restart FF when prompted.
Open Firebug (F12 or the little insect menu to the right of your search box).
In firebug, choose the "Net" tab. Hit "Enable" (text link) to turn it on
Refresh your problem page without using the cache by hitting Ctrl-Shift-R (or Command-shift-R in OSX). You will see the "Net" tab in firefox fill up with a list of each HTTP request made.
Once the page is done loading, hover your mouse over the left colum of each HTTP request shown in the net tab. A tooltip will appear showing you the actual link used. it will be easy to spot any that are http:// instead of https://.
If any of your links resulted in an HTTP redirect, you will see "301 Moved Permanently" in the HTTP status column, and another HTTP request will be just below for the new location. If the problem was due to an external redirect, that's where the evidence will be - the new location's request will be HTTP.
If your problem is due to redirections from an external site, you will see "301 Moved permanently" status codes for the requests that point them to their new location.
Exapnd any of those 301 relocations with the plus sign at the left, and review the response headers to see what is going on. the Location: header will tell you the new location the external server is requesting browsers use.
Make note of this info in the redirect, then send a friendly polite email to the external site in question and ask them to remove the https:// -> http:// redirects for you. Explain how it's breaking the SSL on your site, and ideally include a link to the page that is broken if possible, so that they can see the error for themselves. (this will spur faster action than if you just tell them about the error).
Here is sample output from Firebug for the the external redirect issue.. In my case I found a page calling https:// data feeds was getting the feeds rewritten by the external server to http://.
I've renamed my site to "mysite.example.com" and the external site to "external.example.com", but otherwise left the heders intact. The request headers are shown at the bottom, below the response headers. Note that I"m requesting an https:// link from my site, but getting redirected to an http:// link, which is what was breaking my SSL lock:
Response Headers
Server nginx/0.8.54
Date Fri, 07 Oct 2011 17:35:16 GMT
Content-Type text/html
Content-Length 185
Connection keep-alive
Location http://external.example.com/embed/?key=t6Qu2&width=940&height=300&interval=week&baseAtZero=false
Request Headers
Host external.example.com
User-Agent Mozilla/5.0 (Windows NT 6.1; WOW64; rv:7.0.1) Gecko/20100101 Firefox/7.0.1
Accept */*
Accept-Language en-gb,en;q=0.5
Accept-Encoding gzip, deflate
Accept-Charset ISO-8859-1,utf-8;q=0.7,*;q=0.7
Connection keep-alive
Referer https://mysite.example.com/real-time-data
Cookie JSESSIONID=B33FF1C1F1B732E7F05A547A9CB76ED3
Pragma no-cache
Cache-Control no-cache
So, the important thing to note is that in the Response Headers (above), you are seeing a Location: that starts with http://, not https://. Your browser will take this into account when figuring out if the lock is valid or not, and report only partially encrypted content! (This is actually an important browser security feature to alert users to a potential XSRF and/or phishing attacks).
The solution in this case is not something you can fix on your site - you have to ask the external site to stop their redirect to http. Often this was done on their side for convenience, without realizing this consequence, and a well written, polite email can get it fixed.
For each element loaded in page, check their scheme:
it starts with HTTPS: it is encrypted.
it starts with HTTP: it's not encrypted.
(you can see a relatively complete list on firefox by right-clicking on the page and selecting "View Page Info" then the "medias"tab.
EDIT: FF only shows images and multimedia elements. They are also javascript files & CSS ones which have to be checked. And Firebug is a good tool to find what you need.
Some elements may not list http or https, in this case whichever was used for the page will be used for these items, i.e. if the page request is under SSL then these images will come encrypted while if the page request is not under SSL then these will come unencrypted. Fiddler in Internet Explorer may also be useful in tracking down some of this information.
Sniff the packets - that'll tell you really quick. WireShark is a good program for such a task.
Can firebug do this?
Edit: Looks like firebug will also do this using the "Net" panel, which also gives you some other interesting statistics.
The best tool I have found for detecting http links on a https connection is Fiddler. It's also great for many other troubleshooting efforts.
I use FF plugin HTTPFox for this.
https://addons.mozilla.org/en-us/firefox/addon/httpfox/