Why does alps (or profile) path in spring-data-rest return json body with non matching header Content-Type: text/html? - spring-data-rest

Right now the default Content-Type of my spring-data-rest (spring-boot 1.4.3.RELEASE) provided controllers are application/hal+json which makes sense. If I use chrome I get application/hal+json for the root of my application for instance since chrome uses an Accept header of "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8". However, the /profile (formally /alps) URLs provide text/html even though the response body is json (making the Content-Type not match the body). If you specifically ask for only application/json then you get the correct response header.
Here is the incorrectly working case (returns text/html when the document/body returned is NOT text/html):
$ http --verbose "http://localhost:8080/v1/profile/eldEvents" "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8"
GET /v1/profile/eldEvents HTTP/1.1
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Accept-Encoding: gzip, deflate
Connection: keep-alive
Host: localhost:8080
User-Agent: HTTPie/0.9.2
HTTP/1.1 200
Access-Control-Allow-Credentials: true
Access-Control-Allow-Headers: Origin, X-Requested-With, Content-Type, Accept, Location, X-Auth, Authorization
Access-Control-Allow-Methods: GET, HEAD, POST, PUT, DELETE, TRACE, OPTIONS, PATCH
Access-Control-Allow-Origin: *
Access-Control-Expose-Headers: Location
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
Content-Type: text/html;charset=UTF-8
Date: Fri, 03 Feb 2017 01:16:14 GMT
Expires: 0
Pragma: no-cache
Strict-Transport-Security: max-age=31536000 ; includeSubDomains
Transfer-Encoding: chunked
X-Application-Context: application
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
X-XSS-Protection: 1; mode=block
{
"alps" : {
"version" : "1.0",
"descriptors" : [ {
"id" : "eldEvent-representation",
"href" : "http://localhost:8080/v1/profile/eldEvents",
"descriptors" : [ {
"name" : "sequenceId",
"type" : "SEMANTIC"
}, {
...
Cut out the rest of the response, you can see from above it is json data.
I believe the correct Content-Type for the above request should be something similar to "application/json".

If this is still relevant for you: I've solved this by manually overriding all requests against /profile/* with no content-type defined.
#Component
public class ProfileContentTypeFilter extends OncePerRequestFilter
{
private static final AntPathMatcher matcher = new AntPathMatcher();
#Override
protected void doFilterInternal (HttpServletRequest request, HttpServletResponse response, FilterChain filterChain)
throws ServletException, IOException
{
if (request.getContentType() == null && matcher.match("/profile/*", request.getRequestURI()))
{
// Override response content type for unspecified requests on profile endpoints
response.setContentType(MediaType.APPLICATION_JSON_VALUE);
}
filterChain.doFilter(request, response);
}
}

Related

Google safe browsing API not returning threat URLs

I'm sending requests to the Google safe browsing API. I believe I'm following their documentation correctly. I've tried regenerating my key.
I'm sending the request below
POST https://safebrowsing.googleapis.com/v4/threatMatches:find?key=AIxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx HTTP/1.1
User-Agent: Fiddler
Host: safebrowsing.googleapis.com
Content-Length: 511
{
"client": {
"clientId": "yourcompanyname",
"clientVersion": "1.5.2"
},
"threatInfo": {
"threatTypes": ["MALWARE", "SOCIAL_ENGINEERING"],
"platformTypes": ["WINDOWS"],
"threatEntryTypes": ["URL"],
"threatEntries": [
{"url": "http://www.urltocheck1.org/"},
{"url": "http://malware.testing.google.test"},
{"url": "http://www.urltocheck2.org/"},
{"url": "http://www.urltocheck3.com/"}
]
}
}
And getting back an empty response which is not what I'm expecting with the URLs supplied and following their example.
HTTP/1.1 200 OK
Content-Type: application/json; charset=UTF-8
Date: Wed, 08 Sep 2021 15:05:59 GMT
Server: scaffolding on HTTPServer2
Cache-Control: private
X-XSS-Protection: 0
X-Frame-Options: SAMEORIGIN
X-Content-Type-Options: nosniff
Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-T051=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"
Accept-Ranges: none
Vary: Accept-Encoding
Content-Length: 3
{}
https://transparencyreport.google.com/safe-browsing/search?url=malware.testing.google.test
https://developers.google.com/safe-browsing/v4/lookup-api
You need to pass API key
You need to pass MALWARE url": "http://www.urltocheck1.org/"
if it is not malware it will show empty. try the following url
https://testsafebrowsing.appspot.com/s/malware.html with your code. please search and test with other maleware site

Edge is serving API from cache even after passing cache-control and pragma headers

API is being fetched from cache. This is happening only in Edge. I went through many similar questions in stackoverflow and tried everything but in vain.
I added cache related headers in Vue js
axios.defaults.headers.common['Cache-Control'] = 'private, no-cache, no-store, must-revalidate'
axios.defaults.headers.common['Expires'] = 0
axios.defaults.headers.common['Pragma'] = 'no-cache'
axios.defaults.headers.common['If-Modified-Since'] = 'Mon, 26 Jul 1997 05:00:00 GMT'
I also added cache headers from server side.
[ResponseCache(NoStore = true, Location = ResponseCacheLocation.None)]
[ServiceFilter(typeof(AuthenticateFilter))]
[Produces("application/json")]
[Route("{tenant}")]
public class DashboardController : Controller
{
}
My Request headers looks like in chrome
Accept: application/json, text/plain, */*
Authorization: Token ggggggggggggggggggggg
Cache-Control: private, no-cache, no-store, must-revalidate
Expires: 0
If-Modified-Since: Mon, 26 Jul 1997 05:00:00 GMT
Origin: http://somedummy.com
Pragma: no-cache
Referer: http://somedummy.com/dashboard/sample
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36
withCredentials: true
And response headers :
Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: http://somedummy.com
Cache-Control: no-store,no-cache
Content-Type: application/json; charset=utf-8
Date: Mon, 11 Mar 2019 08:35:36 GMT
Pragma: no-cache
Strict-Transport-Security: max-age=31536000; includeSubDomains;
Transfer-Encoding: chunked
Vary: Origin
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Powered-By: ASP.NET
X-StackifyID: V1|b6841c38-3ec0-4a46-ac24-699ac8a5af0d|
X-XSS-Protection: 1; mode=block
APIs are being fetched from server in IE, chrome and safari but only in Edge it is being served from cache even though "Always refresh from server" option in developer option is selected.
I also have added meta in index.html
<meta http-equiv="expires" content="-1">
<meta http-equiv="cache-control" content="max-age=0">
<meta http-equiv="cache-control" content="no-cache, no-store, must-revalidate">
<meta http-equiv="pragma" content="no-cache">
There are no errors in console. No preflight (OPTIONS) call going from Edge. Strange thing is when fiddler is open then APIs are being served from server in Edge.
Thanks in advance.
Your modification is a Server Response, this won't work, instead you should use headers: { Pragma: 'no-cache' }
Example:
const api = axios.create({
headers: { Pragma: 'no-cache' },
});
Or add it to the configuration
const config = {
headers: { Pragma: 'no-cache'},
params: { id: this.state.taskID }
}
axios.get("some URL", config).then(...)

CSS is not always gzipped why?

In my Firefox or Chrome if I check the HTTP header the result are always with Content-Encoding: gzip. But I have customers reporting that they see "transfer-encoding: chunked" instead and the request are not gzipped.
http://www.example.com/public/css/style.min.css
If I or the customer do a gzip compression online check it's confirmed gzip is active.
https://checkgzipcompression.com = gzip!
But if I use a checker like this one. http://onlinecurl.com/
I also get the transfer-encoding: chunked
Request:
GET /style/css.css HTTP/1.1
Host: www.example.com
Connection: keep-alive
Pragma: no-cache
Cache-Control: no-cache
User-Agent: ...
Accept: /
Referer: http://www.example.com/
Accept-Encoding: gzip, deflate
Accept-Language: ...
Cookie: ...
Response:
HTTP/1.1 200 OK
Age: 532948
cache-control: public, max-age=604800
Content-Type: text/css
Date: Wed, 28 Jun 2017 12:35:07 GMT
ETag: "5349e8d595dfd21:0"
Last-Modified: Wed, 07 Jun 2017 13:56:17 GMT
Server: Microsoft-IIS/7.5
Vary: X-UA,Accept-Encoding, User-Agent
X-Cache: HIT
X-Cache-Hits: 6327
X-CacheReason: Static-js-css.
X-Powered-By: ASP.NET
X-Served-By: ip-xxx-xxx-xxx-xx.name.xxx
x-stale: true
X-UA-Device: pc
X-Varnish: 993020034 905795837
X-Varnish-beresp-grace: 43200.000
X-Varnish-beresp-status: 200
X-Varnish-beresp-ttl: 604800.000
transfer-encoding: chunked
Connection: keep-alive
Why are some requests not gzipped, when it should, this is my Varnish config (the part relevant for gzip):
if (req.http.Accept-Encoding) {
if (req.url ~ "\.(jpg|png|gif|gz|tgz|bz2|tbz|mp3|ogg|flv|swf)$") {
# No point in compressing these
remove req.http.Accept-Encoding;
} elsif (req.http.Accept-Encoding ~ "gzip") {
set req.http.Accept-Encoding = "gzip";
} elsif (req.http.Accept-Encoding ~ "deflate") {
set req.http.Accept-Encoding = "deflate";
} else {
# unkown algorithm
remove req.http.Accept-Encoding;
}
}
# Enabling GZIP
if (beresp.http.Content-Type ~ "(text/css|application/x-javascript|application/javascript)") {
set beresp.do_gzip = true;
}
if (beresp.http.Content-Encoding ~ "gzip" ) {
if (beresp.http.Content-Length == "0") {
unset beresp.http.Content-Encoding;
}
}
set beresp.http.Vary = regsub(beresp.http.Vary, "(?i)^(.*?)X-Forwarded-URI,?(.*)$", "\1\2");
set beresp.http.Vary = regsub(beresp.http.Vary, "(?i)^(.*?)User-Agent,?(.*)$", "\1\2");
set beresp.http.Vary = regsub(beresp.http.Vary, "^(.*?),?$", "X-UA,\1");
set beresp.http.Vary = regsub(beresp.http.Vary, "^(.*?),?$", "\1");
Any ideas, thank you.
Responses will only be gzipped if the request indicates that it can accept a gzipped response. This is indicated by the Accept-Encoding header in the request. So perhaps your online curl is not sending that header. It may be the same for your clients who are seeing this. You really have customers who are reporting that they are not getting responses gzipped?
Update
Ah, I see what you're doing now. Are you using a recent version of Varnish? There's no need to do all this yourself now. Varnish handles it all natively. All you need to do is set do_gzip to on for the content types where you want it, and Varnish takes care of the rest, including the Accept-Encoding header. See the documentation here.
So just remove all of your gzip/encoding related code except the part directly under # Enabling GZIP:
# Enabling GZIP
if (beresp.http.Content-Type ~ "(text/css|application/x-javascript|application/javascript)") {
set beresp.do_gzip = true;
}
And that will probably get everything working. It works fine for me that way. The best amount of VCL is as little as possible, Varnish is very good at handling things itself. Don't forget to restart Varnish or otherwise clear the cache for this site after making the change.
In case it's useful, I use the following VCL for this:
if (
beresp.status == 200
&& beresp.http.content-type ~ "\b((text/(html|plain|css|javascript|xml|xsl))|(application/(javascript|xml|xhtml\+xml)))\b"
) {
set beresp.do_gzip = true;
}
Which checks for more content types that can benefit from compression, including HTML. I don't bother with application/x-javascript as it's ancient and not used.
On another note, are you sure you need to be modifying the Vary header in the way that you are doing there?

How to retrieve a list of Github issues using Postman API

I'm using Postman to test pulling a list of issues from my Github
In command line I can do the following
curl -i -H 'Authorization: token <mytoken>' \
https://api.github.com/user/issues
which I think it returning an empty block:
HTTP/1.1 200 OK
Server: GitHub.com
Date: Tue, 04 Oct 2016 22:01:26 GMT
Content-Type: application/json; charset=utf-8
Content-Length: 5
Status: 200 OK
X-RateLimit-Limit: 5000
X-RateLimit-Remaining: 4988
X-RateLimit-Reset: 1475621615
Cache-Control: private, max-age=60, s-maxage=60
Vary: Accept, Authorization, Cookie, X-GitHub-OTP
ETag: "db74aad51113746c20473c4cf2b08bde"
X-OAuth-Scopes: gist, notifications, repo, user
X-Accepted-OAuth-Scopes:
X-GitHub-Media-Type: github.v3
Access-Control-Expose-Headers: ETag, Link, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval
Access-Control-Allow-Origin: *
Content-Security-Policy: default-src 'none'
Strict-Transport-Security: max-age=31536000; includeSubdomains; preload
X-Content-Type-Options: nosniff
X-Frame-Options: deny
X-XSS-Protection: 1; mode=block
Vary: Accept-Encoding
X-Served-By: cee4c0729c8e9147e7abcb45b9d69689
X-GitHub-Request-Id: 520E2ADF:3E4D:3650D52:57F426B6
[
]
So I am testing it in Postman, but whilst I can do a simple token GET request, i'm getting the following in Postman
{
"message": "Not Found",
"documentation_url": "https://developer.github.com/v3"
}
I had missed adding the scope for the Application Token I had setup
https://developer.github.com/v3/oauth/

The request entity body has an incorrect value in the 'Content-Disposition' header. The expected format for this value is 'Content-Disposition:

I have tried multiple HTTP APIs to post a file to OneDrive using the POST method, and I am getting always the same error.
I went to the extreme case of creating my own SSL TCP socket and send the following bytes:
POST /v5.0/folder.a4fb14adbccd1917.A4FB14ADBCCD1917!32089/files HTTP/1.1
Accept-Encoding:
Authorization: Bearer eWciaQ1DBAAUGCCXc8wU/zFu9QnLdZXy%2bYnElFkAAXA4AJqTmiPoOUADOkV98mAdpBZp8SeF0zjYzU4%2bVa0fVR/aWr4X0cHWAzEF7s7HBEABBptxlWcKHEyrgKfbH2YbADpxEjU0gRrOz37WInvPSgFVD8BZ9GtCQwdHh3GUXunbM/Nlqh1TIzELMJYuPEgaEBBWT25f8SoKziZi2kPVlZDPokjJBq2bxrYcAWydDK74ivuiuDGkX4hMpmWZmWtergEybpN2EgObHQa8O3GT2a9ta2hps0ElCv0GkhWG/u1t19/xjokW2dNbbsJ01A9iJkMYXhaMTYg71sGjqzHdXjAJf0hXNg8NiIOTy82MlHqEwCwEyzfXOSGddt0cLkIDZgAACEDf3skftS5%2bWAHGkGlcwFj4DRlrn3/F2DvOZWgyeITjkWfzdx%2b4B8m5olyO5oYKWf77zjVxnUkJ50cIOVCJq/OgnV1kMGB45EExY3%2b3T9JjN0rm91doGnuFGj1m/vuzVN3eP5f3Jr0hMvXbMpzFBImV/c/5SU5esBzA1zas9xt%2bKEi0rriVSqHS4QI4Bps3RN5KpQd4yrNxcTDi%2b9b9AkUVQB%2bxeW5lWGuR5YvaAOHhcePRqNqT2MY7c1zQMoa/C7E63FuOoSdr0KPJbsDKl8aXEt9vT/6A8BK0MnY%2b7MJ1cZaFL%2bhvK0lfI4Z6LJa/3Ayy%2bPt8%2bOc8gNP1Mu/1RCdOT8K1PxwhnvUIE8cnRyI28LUm%2bWUCnurRCNMNBorhZu64JxqOC1EGOja6wNcyI%2bw3TQ1dorDJWN91qENQAxxBMpl1LWdp548kDJcEPwxI4SkXQKnvWcxW1vCZY90SHzruvNmb
Content-Length: 197
Content-Type: multipart/form-data; boundary=8381f8b9-b470-43ce-b23b-f13cf5840014
Host: apis.live.net
Connection: Keep-Alive
Cache-Control: no-cache
--8381f8b9-b470-43ce-b23b-f13cf5840014
Content-Length: 9
Content-Type: application/octet-stream; charset=UTF-8
Content-Disposition: form-data; name="file"; filename="hello.txt"
xxxxxxxxx
--8381f8b9-b470-43ce-b23b-f13cf5840014--
And I always get the following bytes back:
HTTP/1.1 400 Bad Request
Cache-Control: private, no-cache, no-store, must-revalidate
Content-Length: 276
Content-Type: application/json; charset=UTF-8
Server: Live-API/18.105.8117.3007 Microsoft-HTTPAPI/2.0
P3P: CP="BUS CUR CONo FIN IVDo ONL OUR PHY SAMo TELo"
X-MSNSERVER: BY3301____PAP073
X-Content-Type-Options: nosniff
X-HTTP-Live-Request-Id: API.f06af310-9b49-4cbf-bf71-8c9d79570ec9
X-AsmVersion: UNKNOWN; 18.105.0.0
Date: Mon, 06 Oct 2014 21:37:15 GMT
{
"error": {
"code": "request_body_invalid",
"message": "The request entity body has an incorrect value in the 'Content-Disposition' header. The expected format for this value is 'Content-Disposition: form-data; name=\"file\"; filename=\"[FileName]\"'."
}
}
All of this happened after I tried to answer this SO question.
Any idea?
UPDATE: I used the onedrive-api tag as recommended in the MSDN forums.
Try putting the Content-Disposition header as the first header after the boundary, followed by the Content-Type:
--8381f8b9-b470-43ce-b23b-f13cf5840014
Content-Disposition: form-data; name="file"; filename="hello.txt"
Content-Type: application/octet-stream; charset=UTF-8
You may need to omit the Content-Length as well to make it work. Pretty fragile really.