The priority of Cache-Control:max-age and ETag - browser-cache

I use AWS S3 bucket the save the image files. I add one metadata max-age=315360000 for each file.
The response headers of image file is as below. In the headers both Cache-Control:max-age=315360000 and etag is set.
Cache-Control:max-age=315360000
Date:Thu, 15 Sep 2016 11:03:31 GMT
ETag:"326ae27d85f6d847f819e19955c0f0aa"
Last-Modified:Tue, 06 Sep 2016 03:23:26 GMT
Server:AmazonS3
x-amz-id-2:O/kt/JoZ/hmxSXUWGvkVLudMs6H4MDk8vgS2hIuZvAgGKMdYJsqSiqx/PnpT5AGsJ6FH8O3MGb4=
x-amz-request-id:9289AFE7D3BACB96
x-amz-version-id:P7bZwWZ9PO4I__ZHzwk5tXeiaU6QLSWJ
Question 1: Which one has the highier priority to control the cache?
If the Cache-Control:max-age=315360000 has higher priotity, the response of the next request should be 200 from cache. If the ETag has higher priotity, the response of the next request should be 304.
I make the test on the Mac chrome (Version 53.0.2785.89 (64-bit)). I get a strange behavior.
Test 1 - access the image url to browser. The response is 304.
Test 2 - access an html file which contains the image element to
reference the image. The response is 200 from cache
I use the chrome dev tool to copy the the two requests as curl command lines.
Test 1 curl command lines
curl 'http://s3.cn-north-1.amazonaws.com.cn/xxxx/xxxx.jpg' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.89 Safari/537.36' -H 'Referer: http://pano.shejijia.com/?locale=zh-CN&data=http%3A%2F%2Fjuran-prod-contents.s3.cn-north-1.amazonaws.com.cn%2FAsset%2Fpanos%2F9b57a2b1-0ff7-4c79-ad3d-3343d95f2ad9%2F2702d299-c269-442f-804b-49d94296454e.json&mode=navipano&designId=9b57a2b1-0ff7-4c79-ad3d-3343d95f2ad9' -H 'Origin: http://pano.shejijia.com' --compressed
Test 2 curl command lines
curl 'http://s3.cn-north-1.amazonaws.com.cn/xxxx/xxxx.jpg' -H 'Accept-Encoding: gzip, deflate, sdch' -H 'Accept-Language: en-US,en;q=0.8,zh-CN;q=0.6,zh;q=0.4' -H 'Upgrade-Insecure-Requests: 1' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.89 Safari/537.36' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8' -H 'Cache-Control: max-age=0' -H 'If-None-Match: "d88e24d354fc3b4b43530c0bfe338b01"' -H 'Connection: keep-alive' -H 'If-Modified-Since: Tue, 06 Sep 2016 05:54:16 GMT' --compressed
You can see chrome generates two different curl commands. I guess chrome explains the cache headers in different ways in the two cases.
Questions 2: Why chrome explains the cache headers in different ways in the two cases above?
Questions 3: How to make chrome uses a same manner to response the cache headers? Either one is ok.

Related

curl, API request works in Insomnia, but not in generated code for curl

I have request in Insomnia. When I send it via Insomnia, no problemo. But when I generate source code for curl, it shows me this.
curl --request POST \
--url URL \
--header 'Authorization: bearer XXX' \
--header 'content-type: multipart/form-data; boundary=---011000010111000001101001' \
--cookie JSESSIONID=XXX \
--form datasetFile=#FOLDER/producthierarchyl1-1.zip
{"message":"Bad Request","logref":null,"path":null,"_embedded":{"errors":[{"message":"Required argument [Publisher datasetFile] not specified","logref":null,"path":"/datasetFile","_embedded":{},"_links":{}}]},"_links":{"self":{"href":"/data-api/public-api/v4/datasets/ingestion/","templated":false,"profile":null,"deprecation":null,"title":null,"hreflang":null,"type":null,"name":null}}}
Insomnia output is like this.
> POST PATH HTTP/2
> Host: URL
> user-agent: insomnia/2022.3.0
> cookie: JSESSIONID=XXX
> authorization: bearer XXX
> content-type: multipart/form-data; boundary=X-INSOMNIA-BOUNDARY
> accept: */*
> content-length: 407
* STATE: DO => DID handle 0x170404b14008; line 2077 (connection #27)
* multi changed, check CONNECT_PEND queue!
* STATE: DID => PERFORMING handle 0x170404b14008; line 2196 (connection #27)
| --X-INSOMNIA-BOUNDARY
| Content-Disposition: form-data; name="datasetFile"; filename="producthierarchyl1-1.zip"
| Content-Type: application/zip
| PK�QU�}�%+producthierarchyl1-1.csvUT բ�b
| --X-INSOMNIA-BOUNDARY--
* We are completely uploaded and fine
* HTTP/2 found, allow multiplexing
Where is the catch?
Thanks.
Remove the --header 'content-type: multipart/form-data; boundary=---011000010111000001101001 option.
It really should not be used like this: curl manages that header and the boundary on its own so changing it should only be done in the rare event you truly want and need to fiddle around with it.
A more minor second thing: --request POST should be dropped from the command line, as it is superfluous and easily cause problems if you add --location later on.
Updated
This flaw is said to have been fixed in a later version of Insomnia
For any folks bumping into this issue again, this answer correctly explains the reason for the problem.
Fortunately this should only be reproducible in versions 2022.3.0 and older versions of Insomnia.
If you use the latest one, 2022.5.1, the generated curl will no longer have this problem, example of a multipart request generated:
curl --request GET \
--url http://mockbin.org/request/anything \
--header 'Content-Type: multipart/form-data' \
--form test=test \
--form test=#/Users/yourUserName/Desktop/somefile.txt

Kibana Rest API redirects to login

I'm new to ElasticSearch and Kibana. I'm trying to use Kibana's REST API, but the response i get are redirections to /login whatever the request i try.
The basePath has been modified in kibana.yml :
server.basePath: "/demo"
server.rewriteBasePath: true
Every request, such as :
$ curl -v -u user:passwd -X GET "127.0.0.1:5601/demo/api/features"
gets the following response :
* Hostname was NOT found in DNS cache
* Trying 127.0.0.1...
* Connected to 127.0.0.1 (127.0.0.1) port 5601 (#0)
* Server auth using Basic with user 'user'
> GET /demo/api/features HTTP/1.1
> Authorization: Basic xxxxxxxx
> User-Agent: curl/7.38.0
> Host: 127.0.0.1:5601
> Accept: */*
>
< HTTP/1.1 302 Found
< location: /demo/login?nextUrl=/demo/api/features
< kbn-name: kibana
< content-type: text/html; charset=utf-8
< cache-control: no-cache
< content-length: 0
< connection: close
< Date: Thu, 26 Nov 2020 10:32:56 GMT
<
* Closing connection 0
I'm using v7.2.0 with Linux.
Could someone tell me what i'm doing wrong ?
Best regards
The problem came from the use of the plugin readonlyrest.
First, I had to log in and retrieve the cookie, before querying for anything :
curl -v -X POST "127.0.0.1:5601/demo/login" -d "username=user&password=passwd"-H "kbn-xsrf: true" -H 'Accept: application/json' -c cookie.txt
And then use with the cookie :
curl -v -X GET "127.0.0.1:5601/demo/api/features" -H 'kbn-xsrf: true' -H 'Content-Type: application/json' -H "Accept: application/json" -b cookie.txt
In more recent versions of Kibana and ROR we can obtain the cookie with:
curl '127.0.0.1:5601/login' -H 'kbn-xsrf: true' -H 'Content-Type: application/json' --data-raw '{"username":"user","password":"password"}' -c cookie.txt

Unable to successfully use hydra, http-post-form with Cookie value

I am doing some CTF and stumbled upon a problem with Hydra when trying to use it towards a post request.
I allready know how to solve this, but I am having truble with Hydra.
Intecepted from burp
POST /administrator/index.php HTTP/1.1
Host: 10.10.10.126
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Firefox/68.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Referer: http://10.10.10.126/administrator/index.php
Content-Type: application/x-www-form-urlencoded
Content-Length: 109
Connection: close
Cookie: eaa83fe8b963ab08ce9ab7d4a798de05=0mvnlo2q2792fi1rortceo4or3; 2b01af51830ca9615359108de04d9ca1=bpsnvqrm2mf6hc5akvnr8ftdu1
Upgrade-Insecure-Requests: 1
username=admin&passwd=test&option=com_login&task=login&return=aW5kZXgucGhw&71800a4b802098aac7f3bbd7e49a8665=1
I've been googling a lot and tried several different things without success.
What I did was the following:
hydra -l jonah -P test.txt 10.10.10.126 http-post-form "/administrator/index.php:username=^USER^&passwd=^PASS^&option=com_login&task=login&return=aW5kZXgucGhw&97f7e29809c1606df34ec3df81b10860=1:S=Warning:H=Cookie: eaa83fe8b963ab08ce9ab7d4a798de05=0mvnlo2q2792fi1rortceo4or3; 2b01af51830ca9615359108de04d9ca1=bpsnvqrm2mf6hc5akvnr8ftdu1"
-j is the correct username and
-P test.txt contains the correct password
Could someone tell me what is wrong with my request, and why it doesnt succeed?
There is a bug with version 9.1 of hydra and it won't send a request if you give it a cookie. 9.1 is the version that is packaged with kali still, so you need to upgrade your hydra and it should fix the issue.
Try using like this and replace <IncoorectVerbiage> with something like "invalid password".
hydra -l jonah -P test.txt 10.10.10.126 http-post-form "/administrator/index.php:username=^USER^&passwd=^PASS^&option=com_login&task=login&return=aW5kZXgucGhw&97f7e29809c1606df34ec3df81b10860=1:<IncorrectVerbiage>"

How to repeat https request with curl?

I have the following data captured:
POST /AppHTTP.php HTTP/1.1
Content-Type: application/x-www-form-urlencoded
Content-Length: 89
Host: parkingcab.mos.ru
Connection: Keep-Alive
Accept-Encoding: gzip
User-Agent: App: 4.6() / OS: 7.1.1 / Device: Sony
format=json&appVersion=3&phoneNo=79161234567&PIN=1234&operation=file_list&partnerID=Qulix
HTTP/1.1 200 OK
Date: Sat, 27 Jan 2018 09:57:26 GMT
Server: Apache
Cache-Control: max-age=2592000
Expires: Mon, 26 Feb 2018 09:57:26 GMT
Content-Length: 192
Connection: close
Content-Type: application/json; charset="utf-8"
{"status":1,"operation":"file_list","version":"2.6","response":{"files":[{"file":{"id":"2533922","name":"news.xml","time":"1516885837","type":"text\/xml","partner":"Qulix"}}],"language":"ru"}}
I try to repeat with curl -
curl -vvvv --request POST --header "Content-Type: application/x-www-form-urlencoded" --header "User-Agent: App: 4.6() / OS: 7.1.1 / Device: Sony" --header "Host: pаrkingcаb.mоs.ru" --header "Connection: Keep-Alive" --header "Accept-Encoding: gzip" "https://pаrkingcаb.mоs.ru/AppHTTP.php?format=json&appVersion=3&phoneNo=79161234567&PIN=1234&operation=file_list&partnerID=Qulix"
but I get 503 / No operation response. I also tried to pass data as payload:
curl -vvvv --request POST --header "Content-Type: application/x-www-form-urlencoded" --header "User-Agent: App: 4.6() / OS: 7.1.1 / Device: Sony" --header "Host: pаrkingcаb.mоs.ru" --header "Connection: Keep-Alive" --header "Accept-Encoding: gzip" -d "{'format': 'json', 'appVersion': '3', 'phoneNo': '79161234567', 'PIN': '1234', 'operation': 'file_list', 'partnerID': 'Qulix'}" "https://pаrkingcаb.mоs.ru/AppHTTP.php"
But it didn't help. What am I missing?
You'll like h2c.
If you paste your HTTP request into that, it gives you this curl command line:
curl --header Accept: --compressed --header "Connection: Keep-Alive" --user-agent "App: 4.6() / OS: 7.1.1 / Device: Sony" --data-binary "format=json&appVersion=3&phoneNo=79161234567&PIN=1234&operation=file_list&partnerID=Qulix" https://pаrkingcаb.mоs.ru/AppHTTP.php

strange reference to SignalR [duplicate]

This question already has answers here:
VS2013 RTM making once per second Signal R requests when I check with fiddler
(3 answers)
Closed 9 years ago.
I have an MVC application that I have ported to VS2013 but upon running on my dev machine I have noticed some reference to signalR when I run firebug I don't understand where this come from and why as it seems to be running ever once and then curl 'http://localhost:62899/4645d6c484b844ff9c09255bb97b07b9/arterySignalR/poll?transport=longPolling&connectionToken=AQAAANCMnd8BFdERjHoAwE%2FCl%2BsBAAAAEumWjTMoPkeDOEXv%2BNHRZwAAAAACAAAAAAAQZgAAAAEAACAAAAAh1zpnQcDukz3xmKv6jQ93UejVonzulIb0nfXibU9ozQAAAAAOgAAAAAIAACAAAAC6gz3v487BO50fZx3iXZJKte1SUarM0qPDk9VGtOGWijAAAAD5qwoiK90hXkikTGAOg9nUJzcVRw2x%2BrXg738opqFikHKt4251b6NxJvuH%2F0TKTXhAAAAA51Bi2%2BpR2EmH%2BwbR864v%2Bb0LplFjnpyfFxXvyPrtbB7q7HmbAMmyZbP0HZqGPvxjD%2Fyquhk5vIb4Mjn5VgayGg%3D%3D&messageId=d-45763F85-I%2C0%7CJ%2C4%7CK%2C0&requestUrl=http%3A%2F%2Fmycompany.com%2Fmembership%2Flocation&browserName=Firefox&tid=8&_=1386717058307' -H 'Accept: text/plain, */*; q=0.01' -H 'Accept-Encoding: gzip, deflate' -H 'Accept-Language: en-gb,en;q=0.5' -H 'Content-Type: application/x-www-form-urlencoded; charset=UTF-8' -H 'Host: localhost:62899' -H 'Origin: http://mycompany.com' -H 'Referer: http://mycompany.com/this/that' -H 'User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:25.0) Gecko/20100101 Firefox/25.0'
does anyone know why/what is happenning.
Thanks
This is Browser Link.
It's especially useful with Web Essentials.