curl authentication works with `--netrc` but not `--user` - authentication
With curl 7.74.0, the command
$ curl -b ~/.cookies -c ~/.cookies -L --user 'USERNAME:PASSWORD' https://daac.ornl.gov/daacdata/daymet/Daymet_V4_Stn_Level_CrossVal/data/stnxvalmeta_tmin_pr_2016.txt
yields HTTP 401 (verbose output listed below). However, the following works:
$ echo 'machine urs.earthdata.nasa.gov login USERNAME password PASSWORD' >>~/.netrc
$ curl -b ~/.cookies -c ~/.cookies -L --netrc https://daac.ornl.gov/daacdata/daymet/Daymet_V4_Stn_Level_CrossVal/data/stnxvalmeta_tmin_pr_2016.txt
Why is that? Shouldn't these approaches be equivalent? Perhaps there's an extra option I need to convince curl to pass the credentials set with --user down to urs.earthdata.nasa.gov.
You can create an Earthdata account at https://urs.earthdata.nasa.gov/users/new to test these commands.
(N.B. The --user version works if you already have a valid login cookie from the --netrc version. The question is why login doesn't work.)
The -v output for the --user version is (authenticating cookies etc. censored with "XXX"):
* Trying 160.91.19.24:443...
* Connected to daac.ornl.gov (160.91.19.24) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
* CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
* subject: C=US; ST=Tennessee; L=Oak Ridge; O=Oak Ridge National Laboratory; OU=DAAC; CN=*.ornl.gov
* start date: Jul 14 00:00:00 2020 GMT
* expire date: Oct 11 12:00:00 2022 GMT
* subjectAltName: host "daac.ornl.gov" matched cert's "*.ornl.gov"
* issuer: C=US; O=DigiCert Inc; CN=DigiCert SHA2 Secure Server CA
* SSL certificate verify ok.
* Server auth using Basic with user 'Kodiologist'
> GET /daacdata/daymet/Daymet_V4_Stn_Level_CrossVal/data/stnxvalmeta_tmin_pr_2016.txt HTTP/1.1
> Host: daac.ornl.gov
> Authorization: Basic XXX
> User-Agent: curl/7.74.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 401 Unauthorized
< Date: Mon, 25 Apr 2022 13:02:02 GMT
< Server: Apache
< Strict-Transport-Security: max-age=31536000
< Content-Length: 381
< Content-Type: text/html; charset=iso-8859-1
<
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>401 Unauthorized</title>
</head><body>
<h1>Unauthorized</h1>
<p>This server could not verify that you
are authorized to access the document
requested. Either you supplied the wrong
credentials (e.g., bad password), or your
browser doesn't understand how to supply
the credentials required.</p>
</body></html>
* Connection #0 to host daac.ornl.gov left intact
The -v output for the --netrc version is:
* Couldn't find host daac.ornl.gov in the .netrc file; using defaults
* Trying 160.91.19.24:443...
* Connected to daac.ornl.gov (160.91.19.24) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
* CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
* subject: C=US; ST=Tennessee; L=Oak Ridge; O=Oak Ridge National Laboratory; OU=DAAC; CN=*.ornl.gov
* start date: Jul 14 00:00:00 2020 GMT
* expire date: Oct 11 12:00:00 2022 GMT
* subjectAltName: host "daac.ornl.gov" matched cert's "*.ornl.gov"
* issuer: C=US; O=DigiCert Inc; CN=DigiCert SHA2 Secure Server CA
* SSL certificate verify ok.
> GET /daacdata/daymet/Daymet_V4_Stn_Level_CrossVal/data/stnxvalmeta_tmin_pr_2016.txt HTTP/1.1
> Host: daac.ornl.gov
> User-Agent: curl/7.74.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 302 Found
< Date: Mon, 25 Apr 2022 13:03:08 GMT
< Server: Apache
< Strict-Transport-Security: max-age=31536000
< Location: https://urs.earthdata.nasa.gov/oauth/authorize?app_type=401&client_id=XXXA&response_type=code&redirect_uri=https%3A%2F%2Fdaac.ornl.gov%2Fdaacdata%2Fdoesntmater&state=XXX
< Content-Length: 518
< Content-Type: text/html; charset=iso-8859-1
<
* Ignoring the response-body
* Connection #0 to host daac.ornl.gov left intact
* Issue another request to this URL: 'https://urs.earthdata.nasa.gov/oauth/authorize?app_type=401&client_id=XXX&response_type=code&redirect_uri=https%3A%2F%2Fdaac.ornl.gov%2Fdaacdata%2Fdoesntmater&state=XXX'
* Trying 198.118.243.33:443...
* Connected to urs.earthdata.nasa.gov (198.118.243.33) port 443 (#1)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
* CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
* subject: C=US; ST=Maryland; L=Greenbelt; jurisdictionC=US; O=NASA (National Aeronautics and Space Administration); businessCategory=Government Entity; serialNumber=1958-07-29; CN=urs.earthdata.nasa.gov
* start date: Aug 27 16:09:18 2021 GMT
* expire date: Sep 12 16:09:18 2022 GMT
* subjectAltName: host "urs.earthdata.nasa.gov" matched cert's "urs.earthdata.nasa.gov"
* issuer: C=US; O=Entrust, Inc.; OU=See www.entrust.net/legal-terms; OU=(c) 2014 Entrust, Inc. - for authorized use only; CN=Entrust Certification Authority - L1M
* SSL certificate verify ok.
* Server auth using Basic with user 'Kodiologist'
> GET /oauth/authorize?app_type=401&client_id=XXX&response_type=code&redirect_uri=https%3A%2F%2Fdaac.ornl.gov%2Fdaacdata%2Fdoesntmater&state=XXX HTTP/1.1
> Host: urs.earthdata.nasa.gov
> Authorization: Basic XXX
> User-Agent: curl/7.74.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 302 Found
< Server: nginx/1.20.1
< Date: Mon, 25 Apr 2022 13:03:08 GMT
< Content-Type: text/html; charset=utf-8
< Transfer-Encoding: chunked
< Connection: keep-alive
< X-Frame-Options: SAMEORIGIN
< X-XSS-Protection: 1; mode=block
< X-Content-Type-Options: nosniff
< X-Download-Options: noopen
< X-Permitted-Cross-Domain-Policies: none
< Referrer-Policy: strict-origin-when-cross-origin
< Cache-Control: no-store
< Pragma: no-cache
< Expires: Fri, 01 Jan 1990 00:00:00 GMT
< Access-Control-Allow-Origin: null
< Access-Control-Allow-Credentials: true
< Access-Control-Allow-Methods: GET, POST
< Access-Control-Expose-Headers: true
< Location: https://daac.ornl.gov/daacdata/doesntmater?code=XXX&state=XXX
* Added cookie urs_user_already_logged="yes" for domain earthdata.nasa.gov, path /, expire 1650978188
< Set-Cookie: urs_user_already_logged=yes; domain=earthdata.nasa.gov; path=/; expires=Tue, 26 Apr 2022 13:03:08 GMT; secure; HttpOnly
* Added cookie _urs-gui_session="XXX" for domain urs.earthdata.nasa.gov, path /, expire 1650978188
< Set-Cookie: _urs-gui_session=XXX; path=/; expires=Tue, 26 Apr 2022 13:03:08 GMT; HttpOnly
< X-Request-Id: XXX
< X-Runtime: 0.053834
< Strict-Transport-Security: max-age=31536000
<
* Ignoring the response-body
* Connection #1 to host urs.earthdata.nasa.gov left intact
* Issue another request to this URL: 'https://daac.ornl.gov/daacdata/doesntmater?code=XXX&state=XXX'
* Couldn't find host daac.ornl.gov in the .netrc file; using defaults
* Found bundle for host daac.ornl.gov: XXX [serially]
* Can not multiplex, even if we wanted to!
* Re-using existing connection! (#0) with host daac.ornl.gov
* Connected to daac.ornl.gov (160.91.19.24) port 443 (#0)
> GET /daacdata/doesntmater?code=XXX&state=XXX HTTP/1.1
> Host: daac.ornl.gov
> User-Agent: curl/7.74.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 302 Found
< Date: Mon, 25 Apr 2022 13:03:08 GMT
< Server: Apache
< Strict-Transport-Security: max-age=31536000
* Added cookie AppAuth="XXX" for domain daac.ornl.gov, path /, expire 0
< Set-Cookie: AppAuth=XXX; Path=/
< Location: https://daac.ornl.gov/daacdata/daymet/Daymet_V4_Stn_Level_CrossVal/data/stnxvalmeta_tmin_pr_2016.txt
< Content-Length: 284
< Content-Type: text/html; charset=iso-8859-1
<
* Ignoring the response-body
* Connection #0 to host daac.ornl.gov left intact
* Issue another request to this URL: 'https://daac.ornl.gov/daacdata/daymet/Daymet_V4_Stn_Level_CrossVal/data/stnxvalmeta_tmin_pr_2016.txt'
* Couldn't find host daac.ornl.gov in the .netrc file; using defaults
* Found bundle for host daac.ornl.gov: XXX [serially]
* Can not multiplex, even if we wanted to!
* Connection 0 seems to be dead!
* Closing connection 0
* TLSv1.2 (OUT), TLS alert, close notify (256):
* Hostname daac.ornl.gov was found in DNS cache
* Trying 160.91.19.24:443...
* Connected to daac.ornl.gov (160.91.19.24) port 443 (#2)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
* CApath: /etc/ssl/certs
* SSL re-using session ID
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
* subject: C=US; ST=Tennessee; L=Oak Ridge; O=Oak Ridge National Laboratory; OU=DAAC; CN=*.ornl.gov
* start date: Jul 14 00:00:00 2020 GMT
* expire date: Oct 11 12:00:00 2022 GMT
* subjectAltName: host "daac.ornl.gov" matched cert's "*.ornl.gov"
* issuer: C=US; O=DigiCert Inc; CN=DigiCert SHA2 Secure Server CA
* SSL certificate verify ok.
> GET /daacdata/daymet/Daymet_V4_Stn_Level_CrossVal/data/stnxvalmeta_tmin_pr_2016.txt HTTP/1.1
> Host: daac.ornl.gov
> User-Agent: curl/7.74.0
> Accept: */*
> Cookie: AppAuth=XXX
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Date: Mon, 25 Apr 2022 13:03:09 GMT
< Server: Apache
< Strict-Transport-Security: max-age=31536000
< Last-Modified: Mon, 07 Dec 2020 22:32:53 GMT
< ETag: "5b5-5b5e76833e87e"
< Accept-Ranges: bytes
< Content-Length: 1461
< Vary: Accept-Encoding
< Content-Type: text/plain; charset=UTF-8
<
station_id,station_name,stn_lat,stn_lon,stnz
RQC00660152,AGUIRRE,17.9556,-66.2222,7.6
RQC00664193,GUAYAMA 2E,17.9783,-66.0875,21.9
RQC00665693,MAGUEYES IS,17.9722,-67.0461,3.7
VQC00671740,CHRISTIANSTED FT,17.7469,-64.7014,9.1
VQW00011624,CHRISTIANSTED AP,17.7028,-64.8056,18.6
RQC00660061,ADJUNTAS SUBSTN,18.1747,-66.7978,557.8
RQC00660158,AIBONITO 1 S,18.1281,-66.2642,710.2
RQC00660426,ARECIBO OBSY,18.3494,-66.7525,323.1
RQC00662801,COLOSO,18.3808,-67.1569,12.2
RQC00663431,DOS BOCAS,18.3361,-66.6667,61.0
RQC00664614,HUMACAO NATURAL RESERVE,18.1506,-66.7719,2.7
RQC00664702,ISABELA SUBSTN,18.4653,-67.0525,128.0
RQC00665097,LAJAS SUBSTN,18.0331,-67.0722,27.4
RQC00665807,MANATI 2 E,18.4308,-66.4661,76.2
RQC00665908,MARICAO 2 SSW,18.1511,-66.9889,863.2
RQC00666730,PALMAREJO VEGA BAJA,18.385,-66.43,170.7
RQC00667292,PONCE 4 E,18.0258,-66.5253,21.3
RQC00668306,RIO PIEDRAS EXP STN,18.3906,-66.0542,28.0
RQC00669415,TOA BAJA LEVITTOWN,18.4356,-66.1678,8.5
RQC00669432,TORO NEGRO FOREST,18.1731,-66.4928,868.1
RQC00669521,TRUJILLO ALTO 2 SSW,18.3283,-66.0164,35.1
RQW00011641,SAN JUAN L M MARIN AP,18.4325,-66.0108,2.7
RQC00665064,JUNCOS 1 SE,18.2264,-65.9114,64.9
RQC00666343,CULEBRA HILL,18.2972,-65.29,71.3
RQC00666725,PALMA SOLA,18.3169,-65.8664,466.0
RQC00668814,WFO SAN JUAN,18.4311,-65.9917,3.0
RQW00011630,ROOSEVELT ROADS,18.255,-65.6408,10.1
VQC00677600,REDHOOK BAY ST THOMAS,18.3236,-64.8542,0.6
VQW00011640,CHARLOTTE AMALIE AP,18.3331,-64.9667,6.1
* Connection #2 to host daac.ornl.gov left intact
The --user version works if -L is replaced with --location-trusted and --anyauth is added.
in your netrc you specifically say "this is the username:password for urs.earthdata.nasa.gov", and your command first connects to a different domain: daac.ornl.gov. so your netrc does not trigger on daac.ornl.gov, your "broken" curl command sends the username:password to daac.ornl.gov and and daac.ornl.gov does not like this invalid username:password and your curl request get rejected. meanwhile your working curl command does not send username:password to daac.ornl.gov, then daac.ornl.gov proceed to redirect you to to urs.earthdata.nasa.gov , here your netrc kicks in and your username:password gets sent to urs.earthdata.nasa.gov as it should..
unfortunately curl does not (currently?) support loading netrc from stdin, and does not support setting username:password for specific domains via arguments (not as far as i know anyway?), so your best bet is probably:
NETRCTMPFILE="$(mktemp)"
echo 'machine urs.earthdata.nasa.gov login USERNAME password PASSWORD' > "$NETRCTMPFILE";
curl -b ~/.cookies -c ~/.cookies -L --netrc-file "$NETRCTMPFILE" https://daac.ornl.gov/daacdata/daymet/Daymet_V4_Stn_Level_CrossVal/data/stnxvalmeta_tmin_pr_2016.txt
rm "$NETRCTMPFILE"
You're using quotes around the username and password. In my experience this causes issues.
try
curl -b ~/.cookies -c ~/.cookies -L --user USERNAME:PASSWORD https://daac.ornl.gov/daacdata/daymet/Daymet_V4_Stn_Level_CrossVal/data/stnxvalmeta_tmin_pr_2016.txt
This has issues with your user and password being present in bash history.
This link has a lot of good info on other ways to do this safely too
https://stackoverflow.com/a/56130884/496405
Related
AWS API Gateway MissingAuthenticationToken
I have an api called api-gateway-v1.
this api is a http_proxy.
The method_request authorization is set to NONE.
The method_execution is set to passthrough.
There are two custom domain names which call the same api (same stage).
custom-domain-name-1 has no mutual_tls works as expected.
custom-domain-name-2 has mutual_tls works as expected.
custom domain-1 curl verbose o/p:
curl -v --location --request POST 'https://custom-domain-name-1/v1' --header 'Content-Type: application/json' --data-raw {"email":"email#example.com","password":"mypassword"}'
TCP_NODELAY set
Connected to custom-domain-name-1 (1.2.3.5) port 443 (#0)
ALPN, offering h2
ALPN, offering http/1.1
successfully set certificate verify locations:
CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
TLSv1.3 (OUT), TLS handshake, Client hello (1):
TLSv1.3 (IN), TLS handshake, Server hello (2):
TLSv1.2 (IN), TLS handshake, Certificate (11):
TLSv1.2 (IN), TLS handshake, Server key exchange (12):
TLSv1.2 (IN), TLS handshake, Server finished (14):
TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
TLSv1.2 (OUT), TLS handshake, Finished (20):
TLSv1.2 (IN), TLS handshake, Finished (20):
SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
ALPN, server accepted to use h2
Server certificate:
subject: CN=custom-domain-name-1
start date: Mar 1 00:00:00 2021 GMT
expire date: Mar 30 23:59:59 2022 GMT
subjectAltName: host "custom-domain-name-1" matched cert's "custom-domain-name-1"
issuer: C=US; O=Amazon; OU=Server CA 1B; CN=Amazon
SSL certificate verify ok.
Using HTTP2, server supports multi-use
Connection state changed (HTTP/2 confirmed)
Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
Using Stream ID: 1 (easy handle 0x55c4d244ae10)
POST /v1 HTTP/2
Host: custom-domain-name-1
user-agent: curl/7.68.0
accept: */*
content-type: application/json
content-length: 76
Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
We are completely uploaded and fine
HTTP/2 200
date: Tue, 10 Aug 2021 07:56:41 GMT
content-type: application/json
content-length: 1009
x-amzn-requestid: 8997bed9-b2d5-48fe-8cf2-0d9166ec489d
x-amzn-remapped-connection: keep-alive
x-ratelimit-remaining: 999
x-amz-apigw-id: abc=
cache-control: private, must-revalidate
x-amzn-remapped-server: nginx
x-ratelimit-limit: 1000
x-content-type-options: nosniff
expires: -1
pragma: no-cache
x-amzn-remapped-date: Tue, 10 Aug 2021 07:56:41 GMT
Connection #0 to host custom-domain-name-1 left intact
{"success":{"token":"XYZ"}}
custom domain-2 curl verbose o/p:
curl -v --key form.key --cert cert.pem https://custom-domain-name-2/v1 --header 'Content-Type: application/json' --data-raw '{"email":"email#example.com","password":"mypassword"}'
Note: Unnecessary use of -X or --request, POST is already inferred.
Trying 54.251.193.98:443...
TCP_NODELAY set
Connected to custom-domain-name-2 (1.2.3.4) port 443 (#0)
ALPN, offering h2
ALPN, offering http/1.1
successfully set certificate verify locations:
CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
TLSv1.3 (OUT), TLS handshake, Client hello (1):
TLSv1.3 (IN), TLS handshake, Server hello (2):
TLSv1.2 (IN), TLS handshake, Certificate (11):
TLSv1.2 (IN), TLS handshake, Server key exchange (12):
TLSv1.2 (IN), TLS handshake, Request CERT (13):
TLSv1.2 (IN), TLS handshake, Server finished (14):
TLSv1.2 (OUT), TLS handshake, Certificate (11):
TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
TLSv1.2 (OUT), TLS handshake, CERT verify (15):
TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
TLSv1.2 (OUT), TLS handshake, Finished (20):
TLSv1.2 (IN), TLS handshake, Finished (20):
SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
ALPN, server accepted to use h2
Server certificate:
subject: CN=custom-domain-name-2
start date: Aug 5 00:00:00 2021 GMT
expire date: Sep 3 23:59:59 2022 GMT
subjectAltName: host "custom-domain-name-2" matched cert's "custom-domain-name-2"
issuer: C=US; O=Amazon; OU=Server CA 1B; CN=Amazon
SSL certificate verify ok.
Using HTTP2, server supports multi-use
Connection state changed (HTTP/2 confirmed)
Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
Using Stream ID: 1 (easy handle 0x555588810e10)
POST /v1 HTTP/2
Host: custom-domain-name-2
user-agent: curl/7.68.0
accept: */*
content-type: application/json
content-length: 66
We are completely uploaded and fine
Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
HTTP/2 403
x-amzn-requestid: abc-123
x-amzn-errortype: MissingAuthenticationTokenException
x-amz-apigw-id: 123-abc
content-type: application/json
content-length: 42
date: Wed, 11 Aug 2021 10:53:59 GMT
Connection #0 to host custom-domain-name-2 left intact {"message":"Missing Authentication Token"}
The connection seems to be fine but the HTTP code is changing.
Does anyone has insight into what's happening?
I solved it by Enabling CORS which was missing.
servers returns "Invalid Host header" inside alpine when connecting via ssl
First of all, this question is not about how to fix this, but rather about why it happens.
Exact same curl (exact same version curl/7.65.1) with exact same request produces different results.
Why the certificate chain is different?
How server response depends on ssl connection flow? (Alpine get response body "Invalid host header", host OS downloads file w/o any issues)?
Why http version is different? How do the server and client agree on protocol version? Running the same command with --http curl flag fixes the issue.
docker run -i -t alpine /bin/sh:
apk add curl
curl -kfSL https://nginx.org/download/nginx-1.15.3.tar.gz -o nginx.tar.gz -vvv
Output:
* Trying 95.211.80.227:443...
* TCP_NODELAY set
* Connected to nginx.org (95.211.80.227) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
* subject: CN=localhost
* start date: Jul 8 19:13:11 2019 GMT
* expire date: Aug 7 19:13:11 2019 GMT
* issuer: CN=localhost
* SSL certificate verify result: self signed certificate (18), continuing anyway.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x55e3e4a37540)
> GET /download/nginx-1.15.3.tar.gz HTTP/2
> Host: nginx.org
> User-Agent: curl/7.65.1
> Accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS == 4294967295)!
< HTTP/2 200
< x-powered-by: Express
< content-type: text/html; charset=utf-8
< content-length: 19
< etag: W/"13-OxsTL6IB85fkJxv9HO8uum0slCI"
<
* Connection #0 to host nginx.org left intact
Invalid Host header
Same curl command works completely fine from my host machine (Archlinux bleeding edge) even without insecure (-k)option.
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Trying 95.211.80.227:443...
* TCP_NODELAY set
* Connected to nginx.org (95.211.80.227) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: none
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [108 bytes data]
* TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [2621 bytes data]
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
{ [333 bytes data]
* TLSv1.2 (IN), TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
} [70 bytes data]
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.2 (OUT), TLS handshake, Finished (20):
} [16 bytes data]
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* TLSv1.2 (IN), TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
* subject: CN=nginx.org
* start date: May 14 19:45:30 2019 GMT
* expire date: Aug 12 19:45:30 2019 GMT
* subjectAltName: host "nginx.org" matched cert's "nginx.org"
* issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3
* SSL certificate verify ok.
} [5 bytes data]
> GET /download/nginx-1.15.3.tar.gz HTTP/1.1
> Host: nginx.org
> User-Agent: curl/7.65.1
> Accept: */*
>
{ [5 bytes data]
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Server: nginx/1.15.7
< Date: Wed, 10 Jul 2019 13:03:16 GMT
< Content-Type: application/octet-stream
< Content-Length: 1022881
< Last-Modified: Tue, 28 Aug 2018 15:40:55 GMT
< Connection: keep-alive
< Keep-Alive: timeout=15
< ETag: "5b856d07-f9ba1"
< Accept-Ranges: bytes
P.S. I'm aware of this response
SVN server respond to http range request with full file content, not with range part only
I want to download from SVN server only part of some large file. SVN server is behind Apache HTTP server, so I create HTTP range request. Then got full file content but not requested range part. Server respond with code 200 not 206, so it not supported range requests. How to enable range requests for SVN server?
Is it possible to get only range part from SVN?
What should I change in SVN server config?
May be in Apache server config?
Thanks in advance.
Session log:
request with: Range: bytes=0-199
response with: Accept-Ranges: bytes Content-Length: 678852
$ curl -v -r 0-199 -u l_chayka:***** https://svn2.*******.net/svn/****/tech_research/dvpls/meta.dvpk > meta.dvpk
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Trying 10.128.40.150...
* TCP_NODELAY set
* Connected to svn2.*******.net (10.128.40.150) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: C:/msys64/mingw64/ssl/certs/ca-bundle.crt
CApath: none
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [93 bytes data]
* TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [2580 bytes data]
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
{ [333 bytes data]
* TLSv1.2 (IN), TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
} [70 bytes data]
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.2 (OUT), TLS handshake, Finished (20):
} [16 bytes data]
* TLSv1.2 (IN), TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
* subject: C=CY; L=*****; O=***** Group Limited; CN=*.******.net
* start date: Jun 2 00:00:00 2017 GMT
* expire date: Jul 14 12:00:00 2020 GMT
* subjectAltName: host "svn2.******.net" matched cert's "*.******.net"
* issuer: C=US; O=DigiCert Inc; OU=www.digicert.com; CN=DigiCert SHA2 High Assurance Server CA
* SSL certificate verify ok.
* Server auth using Basic with user 'l_chayka'
} [5 bytes data]
> GET /svn/****/tech_research/dvpls/meta.dvpk HTTP/1.1
> Host: svn2.******.net
> Authorization: Basic bF9aGF5a2Rgd5E6VzI0MTkwODZnMTk4NDE=
> Range: bytes=0-199
> User-Agent: curl/7.62.0
> Accept: */*
>
{ [5 bytes data]
< HTTP/1.1 200 OK
< Date: Mon, 05 Nov 2018 09:50:00 GMT
< Server: Apache/2.4.7 (Ubuntu)
< Strict-Transport-Security: max-age=63072000; includeSubdomains;
< Last-Modified: Thu, 01 Nov 2018 12:29:28 GMT
< ETag: "617837//tech_research/dvpls/meta.dvpk"
< Cache-Control: max-age=604800
< Accept-Ranges: bytes
< Content-Length: 678852
< X-Content-Type-Options: nosniff
< X-Frame-Options: sameorigin
< Content-Type: application/octet-stream
<
{ [5 bytes data]
100 662k 100 662k 0 0 10.2M 0 --:--:-- --:--:-- --:--:-- 10.2M
* Connection #0 to host svn2.******.net left intact
Creating a fulfillment via the Shopify API
I am having trouble creating a fulfillment via the API. I am currently using curl to test, but am receiving a "Required parameter missing" error.
Anybody able to catch the error?
curl command:
curl -i -v -H "Accept: application/json" -X POST -d "{'fulfillment': {'tracking_number': null,'line_items': [{'id': 322959149}]}}" https://[apikey]:[password]#[shop-name].myshopify.com/admin/orders/186122043/fulfillments.json
curl output:
* About to connect() to [shop-name].myshopify.com port 443 (#0)
* Trying 204.93.213.94...
* connected
* Connected to [shop-name].myshopify.com (204.93.213.94) port 443 (#0)
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Server key exchange (12):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using DHE-RSA-AES256-SHA
* Server certificate:
* subject: serialNumber=inVwoUGzj3duFEUBdzDQrbS2kO1qx0Vr; C=CA; ST=Ontario; L=Ottawa; O=Jaded Pixel Technologies Inc.; CN=*.myshopify.com
* start date: 2010-05-10 22:23:08 GMT
* expire date: 2015-08-12 19:17:14 GMT
* subjectAltName: [shop-name].myshopify.com matched
* issuer: C=US; O=Equifax; OU=Equifax Secure Certificate Authority
* SSL certificate verify ok.
* Server auth using Basic with user '4a93b24da162b9ce3f6426206b63e692'
> POST /admin/orders/186122043/fulfillments.json HTTP/1.1
> Authorization: Basic NGE5M2IyNGRhMTYyYjljZTNmNjQyNjIwNmI2M2U2OTI6ZDljMWYwMDNlZWZiMDE1MmQ1MzExMGNjMzk1N2MzMDI=
> User-Agent: curl/7.24.0 (x86_64-apple-darwin12.0) libcurl/7.24.0 OpenSSL/0.9.8y zlib/1.2.5
> Host: [shop-name].myshopify.com
> Accept: application/json
> Content-Length: 76
> Content-Type: application/x-www-form-urlencoded
>
* upload completely sent off: 76 out of 76 bytes
< HTTP/1.1 400 Bad Request
HTTP/1.1 400 Bad Request
< Server: nginx
Server: nginx
< Date: Mon, 07 Oct 2013 21:10:02 GMT
Date: Mon, 07 Oct 2013 21:10:02 GMT
< Content-Type: application/json; charset=utf-8
Content-Type: application/json; charset=utf-8
< Transfer-Encoding: chunked
Transfer-Encoding: chunked
< Status: 400 Bad Request
Status: 400 Bad Request
< X-ShopId: 2235177
X-ShopId: 2235177
< X-Shopify-Asset-Version: 529b6843243276ab3e385c6b38a3bd7b17b4b92f
X-Shopify-Asset-Version: 529b6843243276ab3e385c6b38a3bd7b17b4b92f
< X-Shopify-Shop-Api-Call-Limit: 2/500
X-Shopify-Shop-Api-Call-Limit: 2/500
< HTTP_X_SHOPIFY_SHOP_API_CALL_LIMIT: 2/500
HTTP_X_SHOPIFY_SHOP_API_CALL_LIMIT: 2/500
< X-Frame-Options: DENY
X-Frame-Options: DENY
< X-UA-Compatible: IE=Edge,chrome=1
X-UA-Compatible: IE=Edge,chrome=1
< Set-Cookie: _secure_session_id=989c50f70ca8fd46d381770aa5ad3e72; path=/; secure; HttpOnly
Set-Cookie: _secure_session_id=989c50f70ca8fd46d381770aa5ad3e72; path=/; secure; HttpOnly
< X-Request-Id: 3c394e98aebf3ae3ba4c3f75b8f069be
X-Request-Id: 3c394e98aebf3ae3ba4c3f75b8f069be
<
* Connection #0 to host [shop-name].myshopify.com left intact
{"errors":{"fulfillment":"Required parameter missing"}}
* Closing connection #0
* SSLv3, TLS alert, Client hello (1):
Your ' should "
curl -i -v -H "Accept: application/json" -X POST -d '{"fulfillment": {"tracking_number": null,"line_items": [{"id": 322959149}]}}' https://[apikey]:[password]#[shop-name].myshopify.com/admin/orders/186122043/fulfillments.json
following link should help you.
Github Api v3 giving 404 when I try to create a repo
I've executed the following command with the respective response.
However, if I try to get information about the /user, it works, witch means that my token is valid.
What Am I doing wrong?
guto#willie:~/$ curl -v -XPOST -H 'Authorization: token S3CR3T' -H 'Content-Type: application/json; charset=utf-8' https://api.github.com/user/repos -d '{"name":"my-new-repo","description":"my new repo description"}'
Output:
* About to connect() to api.github.com port 443 (#0)
* Trying 207.97.227.243... connected
* Connected to api.github.com (207.97.227.243) port 443 (#0)
* successfully set certificate verify locations:
* CAfile: none
CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using AES256-SHA
* Server certificate:
* subject: O=*.github.com; OU=Domain Control Validated; CN=*.github.com
* start date: 2009-12-11 05:02:36 GMT
* expire date: 2014-12-11 05:02:36 GMT
* subjectAltName: api.github.com matched
* issuer: C=US; ST=Arizona; L=Scottsdale; O=GoDaddy.com, Inc.; OU=http://certificates.godaddy.com/repository; CN=Go Daddy Secure Certification Authority; serialNumber=07969287
* SSL certificate verify ok.
> POST /user/repos HTTP/1.1
> User-Agent: curl/7.21.3 (x86_64-pc-linux-gnu) libcurl/7.21.3 OpenSSL/0.9.8o zlib/1.2.3.4 libidn/1.18
> Host: api.github.com
> Accept: */*
> Authorization: token S3CR3T
> Content-Type: application/json; charset=utf-8
> Content-Length: 62
>
< HTTP/1.1 404 Not Found
< Server: nginx/1.0.4
< Date: Tue, 27 Dec 2011 03:45:12 GMT
< Content-Type: application/json; charset=utf-8
< Connection: keep-alive
< Status: 404 Not Found
< X-RateLimit-Limit: 5000
< ETag: "31b00b4920d3470b70611b10e0ba62a7"
< X-OAuth-Scopes: public_repo, user
< X-RateLimit-Remaining: 4976
< X-Accepted-OAuth-Scopes: repo
< Content-Length: 29
<
{
"message": "Not Found"
}
* Connection #0 to host api.github.com left intact
* Closing connection #0
* SSLv3, TLS alert, Client hello (1):
guto#willie:~/projetos/apostilas/4linux-helper$
Check the Oath Scope GitHub documentation: $ curl -H "Authorization: bearer TOKEN" https://api.github.com/users/technoweenie -I HTTP/1.1 200 OK X-OAuth-Scopes: repo, user X-Accepted-OAuth-Scopes: user You need to have the repo scope in order to have the right to create a repo, as illustrated by the SO question "Github v3 API - create a REPO". repo DB read/write access, and Git read access to public and private repos. NOTE: Your application can request the scopes in the initial redirection. You can specify multiple scopes by separating them by a comma. https://github.com/login/oauth/authorize? client_id=...& scope=user,public_repo
try using
curl -v -X POST -H 'Content-Type: application/x-www-form-urlencoded' -d '{"name":"my-new-repo","description":"my new repo description"}' https://api.github.com/user/repos?access_token=S3CR3T