I have an AMP story page that even if the AMP Validator passes it as valid, the Google AMP Test prompts me that it is invalid with a Indexing error that says URL marked 'noindex'.
Obviously i do not have any noindex meta on the content of the page and i do not know why the Google AMP Test does not validate the story.
Any clues would be appreciated!
AMP story page
AMP Validator Test
Google AMP Test
This is not caused by the contents of your page but rather the HTTP headers. The headers of this page are:
HTTP/1.1 200 OK =>
Date => Wed, 05 Feb 2020 16:39:25 GMT
Server => Apache
Cache-Control => max-age=0, must-revalidate, private
pragma => no-cache
expires => -1
X-Robots-Tag => noindex
Access-Control-Allow-Origin => *
Access-Control-Allow-Headers => authorization, origin, user-token, x-requested-with, content-type, Content-type
Access-Control-Allow-Methods => PUT, GET, POST, DELETE, OPTIONS
Connection => close
Content-Type => text/html; charset=UTF-8
The X-Robots-Tag => noindex causes Google's AMP Test service to not be able to fetch your page to evaluate whether it is valid. Removing this header should resolve the issue.
Related
I am getting following error
Access to fetch at 'https://mywebsite.com/private-post.html'
(redirected from 'https://api.mywebsite.com/login')
from origin 'https://mywebsite.com' has been blocked by CORS policy:
No 'Access-Control-Allow-Origin' header is present on the requested resource.
If an opaque response serves your needs,
set the request's mode to 'no-cors' to fetch the resource with CORS disabled.
I have following cors preflight set to https://api.mywebsite.com. https://mywebsite.com is static website. I am not sure if it has cors.
Access-Control-Allow-Origin: ['https://mywebsite.com', 'https://api.mywebsite.com']
Access-Control-Allow-Headers: [*]
Access-Control-Allow-Methods: [DELETE, GET, OPTION, PUT, HEAD, PATCH, POST]
Access-Control-Allow-Credentials: true
So first I make post request from https://mywebsite.com/login.html to https://api.mywebsite.com/login endpoint. And after successuful login I am doing following redirect with lambda. While having above cors. And I cannot redirect to any page because of error above.
const response = {
statusCode: 303,
headers: {
'Location': 'https://mywebsite.com/private-post.html'
}
}
return response
Edit
I tried setting Access-Control-Allow-Origin: [*] but then I cannot set Access-Control-Allow-Credentials: true.
So to fix that I set Access-Control-Allow-Origin: ['https://*'] now I can allow credentials however on redirect the cors error still remains the same.
https://mywebsite.com is static website. I am not sure if it has cors.
The error message days it doesn’t but, since you are trying to read the content with JS, it needs to.
You can’t use a redirect to grant CORS permission for a resource you don’t control. If you could, then an attacker could (for example) create a site that redirects to your online banking, and then have JS on their attacking site access your accounts through it. It would make the Same Origin Policy worthless.
General:
Request URL:x/site.php
Request Method:OPTIONS
Status Code:302 Found
Remote Address:x.x.x.x:80
Response Headers:
view source
Access-Control-Allow-Headers:Content-Type
Access-Control-Allow-Origin:*
Access-Control-Max-Age:300
Cache-Control:no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Content-Length:0
Content-Type:text/html; charset=UTF-8
Date:Thu, 02 Mar 2017 14:27:21 GMT
Expires:Thu, 19 Nov 1981 08:52:00 GMT
Location:y
Pragma:no-cache
Server:Apache/2.4.25 (Ubuntu)
Request Headers:
view source
Accept:*/*
Accept-Encoding:gzip, deflate, sdch
Accept-Language:en-US,en;q=0.8
Access-Control-Request-Headers:authorization
Access-Control-Request-Method:POST
Cache-Control:no-cache
Connection:keep-alive
DNT:1
Host:x
Origin:http://127.0.0.1:3000
Pragma:no-cache
Referer:http://127.0.0.1:3000/
User-Agent:Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.90 Safari/537.36
Apache virtualhost config looks as so:
<IfModule mod_headers.c>
Header set Access-Control-Allow-Origin "http://127.0.0.1:3000"
Header set Access-Control-Allow-Origin "http://127.0.0.1"
Header set Access-Control-Max-Age "300"
Header set Access-Control-Allow-Credentials "true"
Header set Access-Control-Allow-Headers "Origin, X-Requested-With, Content-Type, Accept"
Header set Access-Control-Allow-Methods "POST, GET, PUT, DELETE, PATCH, OPTIONS"
</IfModule>
The preflight request is skipping the apache config and hitting my webapp directly, which does a redirect (hence the 302 and the location: y).
I don't know why the preflight request is not being handled by apache?
To fully CORS-enable an Apache web server, you need to have it configured to look like this:
Header always set Access-Control-Allow-Origin "*"
Header always set Access-Control-Allow-Headers "Authorization"
Header always set Access-Control-Allow-Methods "GET"
Header always set Access-Control-Expose-Headers "Content-Security-Policy, Location"
Header always set Access-Control-Max-Age "600"
RewriteEngine On
RewriteCond %{REQUEST_METHOD} OPTIONS
RewriteRule ^(.*)$ $1 [R=200,L]
Longer explanation at https://benjaminhorn.io/code/setting-cors-cross-origin-resource-sharing-on-apache-with-correct-response-headers-allowing-everything-through/
Some general notes on what values to set for the various Access-Control- response headers:
Access-Control-Allow-Headers: you must set it to include any header names your request sends except CORS-safelisted header names or so-called “forbidden” header names (names of headers set by the browser that you can’t set in your JavaScript); the spec alternatively allows the * wildcard as its value—so you can try it, though some browsers may not support it yet: Chrome bug, Firefox bug, Safari bug.
Access-Control-Allow-Methods: the spec alternatively allows the * wildcard—but again, as with Access-Control-Allow-Headers: *, some browsers may not support it yet.
Access-Control-Expose-Headers: set to include any response headers beyond Expires, Cache-Control, Content-Type, Pragma, Last-Modified, and Content-Language that your frontend code needs to read. A lot of people forget to set this and end up baffled about why they can’t read the value of a particular response header). Again the spec alternatively allows the * wildcard here, but some browsers may not support it yet.
Access-Control-Max-Age: Chrome has an upper limit of 600 (10 minutes) hardcoded, so there’s no point in setting a higher value for it than that (Chrome will just throttle it down to 10 minutes if you set it higher, and Safari limits it to only 5 minutes).
So then, about the particular request shown in the question, the specific changes and additions that would need to made are these:
Use Header always set instead of just Header set.
Use mod_rewrite to handle the OPTIONS by just sending back 200 OK with those headers.
The request has Access-Control-Request-Headers:authorization so in the Apache config, add Authorization in the Access-Control-Allow-Headers response header too.
Origin is a “forbidden” header name set by the browser, and Accept is a CORS-safelisted header name, so no need to include them in Access-Control-Allow-Headers.
The request sends no Content-Type, so no need for it in Access-Control-Allow-Headers in the response (and never needed for GET requests and otherwise only needed if the type is not application/x-www-form-urlencoded, text/plain, or multipart/form-data).
For Access-Control-Allow-Methods, the request seems to just be a GET, so unless the plan’s to also make POST/PUT/DELETE/PATCH requests, no point in including them.
I have a CloudFront distribution that blocks the font download in Chrome (desktop version) with the fallowing error:
Font from origin 'https://....cloudfront.net' has been blocked from
loading by Cross-Origin Resource Sharing policy: No
'Access-Control-Allow-Origin' header is present on the requested
resource. Origin 'https://example.com' is therefore not allowed
access.
Where should I set this Access-Control-Allow-Origin header?
I tried adding the header in the "Origin" section of the could distribution but it does not produce any effect.
EDIT #1:
Nginx configuration on origin has the fallowing directive:
location ~ \.(eot|ttf|woff|woff2)$ {
add_header Access-Control-Allow-Origin *;
}
which on this test curl -I https://example.com/skin/frontend/smartwave/default/megamenu/css/fonts/fontawesome-webfont.woff
Retuns the fallowing response:
HTTP/1.1 200 OK
Server: nginx
Date: Tue, 02 Feb 2016 17:53:39 GMT
Content-Type: application/font-woff
Content-Length: 44432
Last-Modified: Wed, 13 May 2015 15:58:11 GMT
Connection: keep-alive
ETag: "55537493-ad90"
Pragma: public
Cache-Control: max-age=31536000, public, must-revalidate, proxy-revalidate
Accept-Ranges: bytes
From what I see here this header Access-Control-Allow-Origin is missing.
Also I whitelisted the header on CloudFront so that it will not block it:
That was hard to trace as the rules for headers were set in 2 different locations and not in one.
Fixing the correct header for the correct type of file did the job, but you have to consider the fact that in some locations trying to overwrite the NGINX rules does not work. It will only consider the first rule.
A comprehensive description of headers can be found here https://stackoverflow.com/a/10636765/1168944
Given these request headers:
Host: api.example.org
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:41.0) Gecko/20100101 Firefox/41.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
DNT: 1
Origin: https://web.example.org
Access-Control-Request-Method: GET
Access-Control-Request-Headers: authorization
Connection: keep-alive
Pragma: no-cache
Cache-Control: no-cache
And these response headers:
Connection: keep-alive
Content-Length: 0
Content-Type: text/plain; charset=utf-8
Date: Tue, 13 Oct 2015 10:57:34 GMT
Server: nginx/1.8.0
access-control-allow-headers: Authorization, Content-Type
access-control-allow-methods: PUT, DELETE, PATCH
access-control-allow-origin: *
This works even though only the Authorization and Content-Type headers are explicitly allowed. Why didn't I have to allow other headers that my browser sends? (like DNT for example)
Update: this MDN page contains an overview of simple headers (default CORS-safelisted request headers):
A simple header (or CORS-safelisted request header) is one of the
following HTTP headers:
Accept
Accept-Language
Content-Language
Content-Type with a MIME type of its parsed value (ignoring parameters) of either application/x-www-form-urlencoded, multipart/form-data, or text/plain.
Or one of these client hint headers:
DPR
Downlink
Save-Data
Viewport-Width
Width
Without seeing your code to generate the headers, or on which system you are serving from, i.e. nginx or apache, the best I can do is refer you to http://client.cors-api.appspot.com/client which will allow you to test your CORS requests. Also, you should look at http://enable-cors.org/server.html for your specific setup. For instance on nginx, you could have something like this
add_header 'Access-Control-Allow-Headers' 'DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type';
There is a set of normal headers, and then the set of headers that you have to explicitly call out. see http://www.html5rocks.com/en/tutorials/cors/#toc-adding-cors-support-to-the-server about setting it up on a server.
The Access-Control-Allow-Headers is attached by backend, you cannot control that header on client side.Access-Control-Allow-Headers should be returned in response object.
So to include other headers into Access-Control-Allow-Headers header in response object - you have to configure your web server or update backend application which serves requests to attach desired value of Access-Control-Allow-Headers to each request.
To allow any headers in your client requests server should add Access-Control-Allow-Origin: * header to each response.
There are a lot of articles and info of how you can setup CORS to work in the way you want. For example that one - Enabling CORS
I am starting to learn about http correctly.
I am working in lamp stack.
On the command line i am requesting a local page which will be served with apache to see the headers that are returned.
curl -i local.testsite
The page i am requesting has no content and i am not setting any headers but there are already a lot of headers sent in the response such as:
HTTP/1.1 200 OK
Date: Thu, 17 Jan 2013 20:28:52 GMT
Server: Apache/2.2.22 (Ubuntu)
X-Powered-By: PHP/5.3.10-1ubuntu3.4
Vary: Accept-Encoding
Content-Length: 0
Content-Type: text/html
So if i am not setting these, does apache set these automatically?
Yes Apache is setting those by default. By the way, if you only care about the headers, you should use
curl -I local.testsite
-I returns the headers only (HTTP HEAD request), such that even if you had content on the page you would only get the header.
Some are set by PHP:
The X-Powered-By header is set by the expose_php INI setting.
The Content-Type header is set by the default_mimetype INI setting.
The others are set by Apache:
The Server header is set by the ServerSignature directive.
The Vary: Accept-Encoding header is usually sent when mod_deflate is enabled.
Date and Content-Length are not configurable as they are part of the HTTP spec. Date is included as a MUST (except under some conditions) and Content-Length as a SHOULD.
See also How to remove date header from apache? and How to disable the Content-Length response header with Apache?.