Why can't googlebot fetch my JavaScript file? - http-headers

https://api-staging-weld.freetls.fastly.net/scripts/customdomain_weld.19f3e9ec.js
The error I get in Google Search Console is Temporarily unreachable with no further explanation. I think it's something with my http headers but can't figure out which one. Here they are:
accept-ranges: bytes
access-control-allow-origin: *
age: 4905
cache-control: public, max-age=31536000
content-encoding: gzip
content-length: 29990
content-type: application/javascript; charset=UTF-8
date: Fri, 20 Apr 2018 12:46:37 GMT
etag: W/"1a018-162e269bfe0"
last-modified: Fri, 20 Apr 2018 09:36:44 GMT
server: Cowboy
status: 200
vary: Accept-Encoding
via: 1.1 vegur
via: 1.1 varnish
x-cache: HIT
x-cache-hits: 1
x-powered-by: Express
x-served-by: cache-bma7030-BMA
x-timer: S1524228398.925110,VS0,VE6
What's wrong or what am i Missing?

Related

CORS issues with CloudFront and S3

Some of our users are getting this error in the browser consoles:
XMLHttpRequest cannot load https://d2ynz9y5qgba7.cloudfront.net/icons/instagram.svg.
No 'Access-Control-Allow-Origin' header is present on the requested resource.
Origin 'https://www.weld.io'; is therefore not allowed access.
These are the current headers:
Accept-Ranges: bytes
Access-Control-Allow-Methods: GET
Access-Control-Allow-Origin: *
Age: 21038
Content-Length: 1086
Content-Type: image/svg+xml
Date: Wed, 23 Mar 2016 10:00:12 GMT
Etag: "5cc3b790e15d7736b95880001521630b"
Last-Modified: Wed, 18 Mar 2015 10:20:32 GMT
Server: AmazonS3
Via: 1.1 8cdc69e06e564b9aef153cf0b52204b0.cloudfront.net (CloudFront)
X-Amz-Cf-Id: 0lD_0czYTm6i5f7RlURkG2dcQx_B252LC2GOu98anzHH1k1eaz2lmg==
X-Cache: Hit from cloudfront
As you can see, Access-Control-Allow-Origin is indeed present.
We are also "whitelisting" Origin on CloudFront.
What could be the cause of this issue?

HHVM + Apache + Varnish + Drupal 404

This seems like quite an odd thing to be happening, but I'm getting 404 responses but the pages are still displaying as expected.
I do have a slightly odd setup on this server, as we're running HHVM for PHP pages and using Varnish as we need to direct some of the pages to our old server.
We're running Drupal on this server and it seems to work fine except the 404 response seems to be stopping the login form from working.
I was going to add some images to show what's going on, but unfortunately don't have enough reputation....
here's the what we get from a GET -Sed request
pete#pete-work ~ $ GET -Sed http://beta.newint.org/user
GET http://beta.newint.org/user
404 Not Found
Cache-Control: no-cache, must-revalidate, post-check=0, pre-check=0
Connection: close
Date: Wed, 27 May 2015 15:14:23 GMT
Via: 1.1 varnish
Age: 0
Server: Apache
Vary: Accept-Encoding
Content-Language: en
Content-Type: text/html; charset=utf-8
Expires: Sun, 19 Nov 1978 05:00:00 GMT
Client-Date: Wed, 27 May 2015 15:14:23 GMT
Client-Peer: 178.79.141.247:80
Client-Response-Num: 1
Client-Transfer-Encoding: chunked
ImageToolbar: false
Link: <http://beta.newint.org/user>; rel="canonical",<http://beta.newint.org/user>; rel="shortlink"
Title: User account | Site-Install
X-Generator: Drupal 7 (http://drupal.org)
X-Meta-Charset: utf-8
X-Meta-Generator: Drupal 7 (http://drupal.org)
X-Meta-Viewport: width=device-width, maximum-scale = 1.0
X-Powered-By: HHVM/3.7.0
X-Varnish: 1786764394
And then bypassing varnish and going straight to apache
pete#pete-work ~ $ GET -Sed http://beta.newint.org:8080/user
GET http://beta.newint.org:8080/user
404 Not Found
Cache-Control: no-cache, must-revalidate, post-check=0, pre-check=0
Connection: close
Date: Wed, 27 May 2015 15:14:31 GMT
Server: Apache
Vary: Accept-Encoding
Content-Language: en
Content-Type: text/html; charset=utf-8
Expires: Sun, 19 Nov 1978 05:00:00 GMT
Client-Date: Wed, 27 May 2015 15:14:32 GMT
Client-Peer: 178.79.141.247:8080
Client-Response-Num: 1
Client-Transfer-Encoding: chunked
ImageToolbar: false
Link: <http://beta.newint.org:8080/user>; rel="canonical",<http://beta.newint.org:8080/user>; rel="shortlink"
Title: User account | Site-Install
X-Generator: Drupal 7 (http://drupal.org)
X-Meta-Charset: utf-8
X-Meta-Generator: Drupal 7 (http://drupal.org)
X-Meta-Viewport: width=device-width, maximum-scale = 1.0
X-Powered-By: HHVM/3.7.0
Any ideas?
Turns out this was simply due to mod_rewrite not being enabled.

Webmaster tools do not show a website as verified using Site Verification API

Why Google webmaster tool doesn't show the verified site on the webmaster tool? I am getting status code 200 when using both OAuth 2 playground and my program. But when check on https://www.google.com/webmasters/tools/ I cannot see the verified site.
This is the response I am getting from Google:
HTTP/1.1 200 OK
Content-length: 220
Via: HTTP/1.1 GWA
X-content-type-options: nosniff
Etag: "00deUEyKiunbfrZpRY_GOwanRBXo/rPzr3x8uGF7cZ_o8tWZ9Z9Yp87RwQU"
X-google-cache-control: remote-fetch
-content-encoding: gzip
Server: GSE
X-xss-protection: 1; mode=block
Pragma: no-cache
Cache-control: no-cache, no-store, max-age=0, must-revalidate
Date: Wed, 10 Jul 2013 09:38:26 GMT
X-frame-options: SAMEORIGIN
Content-type: application/json; charset=UTF-8
Expires: Fri, 01 Jan 1990 00:00:00 GMT
{"id":"http%3A%2F%2Fsample.net%2F","site":{"type":"SITE","identifier":"http://sample.net/"},"owners":["sample#gmail.com"]}
Can anyone suggest the reason behind this?

GDATA and youtube, Get all videos on a playlist

Does anyone know how to get all the videos from a playlist ?
E.g from this playlist
https://gdata.youtube.com/feeds/api/playlists/454A0274C1223C90?v=2
it caps at 10 results
I tried &max-results=1000 but that 404'ed
try asking for 50 results at a time ( max allowed )
GET /feeds/api/playlists/454A0274C1223C90?v=2&&max-results=100&fields=entry/id,entry/title&key=4{your-key} HTTP/1.1
Host: gdata.youtube.com
Authorization: OAuth ya29.AHES6ZQMwDjrUKh8RLysv-ZzF3DaUmg0XmG0jtVcWdSA0A
HTTP/1.1 400 Bad Request
status: 400
content-length: 246
x-xss-protection: 1; mode=block
x-content-type-options: nosniff
expires: Sat, 28 Apr 2012 16:29:01 GMT
x-google-cache-control: remote-fetch
server: GSE
via: HTTP/1.1 GWA
x-gdata-user-country: US
cache-control: private, max-age=0
date: Sat, 28 Apr 2012 16:29:01 GMT
x-frame-options: SAMEORIGIN
content-type: application/vnd.google.gdata.error+xml
-content-encoding: gzip
<errors xmlns='http://schemas.google.com/g/2005'><error><domain>GData</domain><code>InvalidRequestUriException</code><internalReason>Max-results value is too high. Only up to 50 results can be returned per query.
GET /feeds/api/playlists/454A0274C1223C90?v=2&&max-results=50&fields=entry/id,entry/title&key=${key} HTTP/1.1
Host: gdata.youtube.com
Authorization: OAuth ya29.AHES6ZQMwDjrUKh8RLysv-ZzF3DaUmg0XmG0jtVcWdSA0A
HTTP/1.1 200 OK
status: 200
gdata-version: 2.1
content-length: 6703
via: HTTP/1.1 GWA
content-location: https://gdata.youtube.com/feeds/api/playlists/454A0274C1223C90?v=2&&max-results=50&fields=entry/id,entry/title&key=w1Q
x-content-type-options: nosniff
expires: Sat, 28 Apr 2012 16:32:08 GMT
vary: Accept, X-GData-Authorization, GData-Version
x-google-cache-control: remote-fetch
server: GSE
last-modified: Sat, 28 Apr 2012 10:46:07 GMT
x-xss-protection: 1; mode=block
etag: W/"A04CRn47eCp7I2A9WhVWFk0."
cache-control: private, max-age=300, no-transform
date: Sat, 28 Apr 2012 16:32:08 GMT
x-frame-options: SAMEORIGIN
content-type: application/atom+xml; charset=UTF-8; type=feed
-content-encoding: gzip
x-gdata-user-country: US

Why won't this cache?

HTTP/1.1 200 OK
Cache-Control: private
Content-Type: application/xml; charset=utf-8
Expires: Tue, 06 Dec 2011 21:42:35 GMT
ETag: 28
Vary: *
Server: Microsoft-IIS/7.5
X-AspNet-Version: 2.0.50727
X-Powered-By: ASP.NET
Date: Tue, 06 Dec 2011 21:42:24 GMT
Content-Length: 258
I am building the expiration date myself and setting it, yet it still won't cache. I am using SqlDependency with this, but what I want for it to do is hold it in cache for say...30 minutes, then after 30 sql cache comes into play.
First thing that I notice is "Cache-Control: private". As long is it private it won't be cached by proxies and I'm not sure all the browsers will cache it.