With curl 7.74.0, the command
$ curl -b ~/.cookies -c ~/.cookies -L --user 'USERNAME:PASSWORD' https://daac.ornl.gov/daacdata/daymet/Daymet_V4_Stn_Level_CrossVal/data/stnxvalmeta_tmin_pr_2016.txt
yields HTTP 401 (verbose output listed below). However, the following works:
$ echo 'machine urs.earthdata.nasa.gov login USERNAME password PASSWORD' >>~/.netrc
$ curl -b ~/.cookies -c ~/.cookies -L --netrc https://daac.ornl.gov/daacdata/daymet/Daymet_V4_Stn_Level_CrossVal/data/stnxvalmeta_tmin_pr_2016.txt
Why is that? Shouldn't these approaches be equivalent? Perhaps there's an extra option I need to convince curl to pass the credentials set with --user down to urs.earthdata.nasa.gov.
You can create an Earthdata account at https://urs.earthdata.nasa.gov/users/new to test these commands.
(N.B. The --user version works if you already have a valid login cookie from the --netrc version. The question is why login doesn't work.)
The -v output for the --user version is (authenticating cookies etc. censored with "XXX"):
* Trying 160.91.19.24:443...
* Connected to daac.ornl.gov (160.91.19.24) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
* CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
* subject: C=US; ST=Tennessee; L=Oak Ridge; O=Oak Ridge National Laboratory; OU=DAAC; CN=*.ornl.gov
* start date: Jul 14 00:00:00 2020 GMT
* expire date: Oct 11 12:00:00 2022 GMT
* subjectAltName: host "daac.ornl.gov" matched cert's "*.ornl.gov"
* issuer: C=US; O=DigiCert Inc; CN=DigiCert SHA2 Secure Server CA
* SSL certificate verify ok.
* Server auth using Basic with user 'Kodiologist'
> GET /daacdata/daymet/Daymet_V4_Stn_Level_CrossVal/data/stnxvalmeta_tmin_pr_2016.txt HTTP/1.1
> Host: daac.ornl.gov
> Authorization: Basic XXX
> User-Agent: curl/7.74.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 401 Unauthorized
< Date: Mon, 25 Apr 2022 13:02:02 GMT
< Server: Apache
< Strict-Transport-Security: max-age=31536000
< Content-Length: 381
< Content-Type: text/html; charset=iso-8859-1
<
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>401 Unauthorized</title>
</head><body>
<h1>Unauthorized</h1>
<p>This server could not verify that you
are authorized to access the document
requested. Either you supplied the wrong
credentials (e.g., bad password), or your
browser doesn't understand how to supply
the credentials required.</p>
</body></html>
* Connection #0 to host daac.ornl.gov left intact
The -v output for the --netrc version is:
* Couldn't find host daac.ornl.gov in the .netrc file; using defaults
* Trying 160.91.19.24:443...
* Connected to daac.ornl.gov (160.91.19.24) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
* CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
* subject: C=US; ST=Tennessee; L=Oak Ridge; O=Oak Ridge National Laboratory; OU=DAAC; CN=*.ornl.gov
* start date: Jul 14 00:00:00 2020 GMT
* expire date: Oct 11 12:00:00 2022 GMT
* subjectAltName: host "daac.ornl.gov" matched cert's "*.ornl.gov"
* issuer: C=US; O=DigiCert Inc; CN=DigiCert SHA2 Secure Server CA
* SSL certificate verify ok.
> GET /daacdata/daymet/Daymet_V4_Stn_Level_CrossVal/data/stnxvalmeta_tmin_pr_2016.txt HTTP/1.1
> Host: daac.ornl.gov
> User-Agent: curl/7.74.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 302 Found
< Date: Mon, 25 Apr 2022 13:03:08 GMT
< Server: Apache
< Strict-Transport-Security: max-age=31536000
< Location: https://urs.earthdata.nasa.gov/oauth/authorize?app_type=401&client_id=XXXA&response_type=code&redirect_uri=https%3A%2F%2Fdaac.ornl.gov%2Fdaacdata%2Fdoesntmater&state=XXX
< Content-Length: 518
< Content-Type: text/html; charset=iso-8859-1
<
* Ignoring the response-body
* Connection #0 to host daac.ornl.gov left intact
* Issue another request to this URL: 'https://urs.earthdata.nasa.gov/oauth/authorize?app_type=401&client_id=XXX&response_type=code&redirect_uri=https%3A%2F%2Fdaac.ornl.gov%2Fdaacdata%2Fdoesntmater&state=XXX'
* Trying 198.118.243.33:443...
* Connected to urs.earthdata.nasa.gov (198.118.243.33) port 443 (#1)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
* CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
* subject: C=US; ST=Maryland; L=Greenbelt; jurisdictionC=US; O=NASA (National Aeronautics and Space Administration); businessCategory=Government Entity; serialNumber=1958-07-29; CN=urs.earthdata.nasa.gov
* start date: Aug 27 16:09:18 2021 GMT
* expire date: Sep 12 16:09:18 2022 GMT
* subjectAltName: host "urs.earthdata.nasa.gov" matched cert's "urs.earthdata.nasa.gov"
* issuer: C=US; O=Entrust, Inc.; OU=See www.entrust.net/legal-terms; OU=(c) 2014 Entrust, Inc. - for authorized use only; CN=Entrust Certification Authority - L1M
* SSL certificate verify ok.
* Server auth using Basic with user 'Kodiologist'
> GET /oauth/authorize?app_type=401&client_id=XXX&response_type=code&redirect_uri=https%3A%2F%2Fdaac.ornl.gov%2Fdaacdata%2Fdoesntmater&state=XXX HTTP/1.1
> Host: urs.earthdata.nasa.gov
> Authorization: Basic XXX
> User-Agent: curl/7.74.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 302 Found
< Server: nginx/1.20.1
< Date: Mon, 25 Apr 2022 13:03:08 GMT
< Content-Type: text/html; charset=utf-8
< Transfer-Encoding: chunked
< Connection: keep-alive
< X-Frame-Options: SAMEORIGIN
< X-XSS-Protection: 1; mode=block
< X-Content-Type-Options: nosniff
< X-Download-Options: noopen
< X-Permitted-Cross-Domain-Policies: none
< Referrer-Policy: strict-origin-when-cross-origin
< Cache-Control: no-store
< Pragma: no-cache
< Expires: Fri, 01 Jan 1990 00:00:00 GMT
< Access-Control-Allow-Origin: null
< Access-Control-Allow-Credentials: true
< Access-Control-Allow-Methods: GET, POST
< Access-Control-Expose-Headers: true
< Location: https://daac.ornl.gov/daacdata/doesntmater?code=XXX&state=XXX
* Added cookie urs_user_already_logged="yes" for domain earthdata.nasa.gov, path /, expire 1650978188
< Set-Cookie: urs_user_already_logged=yes; domain=earthdata.nasa.gov; path=/; expires=Tue, 26 Apr 2022 13:03:08 GMT; secure; HttpOnly
* Added cookie _urs-gui_session="XXX" for domain urs.earthdata.nasa.gov, path /, expire 1650978188
< Set-Cookie: _urs-gui_session=XXX; path=/; expires=Tue, 26 Apr 2022 13:03:08 GMT; HttpOnly
< X-Request-Id: XXX
< X-Runtime: 0.053834
< Strict-Transport-Security: max-age=31536000
<
* Ignoring the response-body
* Connection #1 to host urs.earthdata.nasa.gov left intact
* Issue another request to this URL: 'https://daac.ornl.gov/daacdata/doesntmater?code=XXX&state=XXX'
* Couldn't find host daac.ornl.gov in the .netrc file; using defaults
* Found bundle for host daac.ornl.gov: XXX [serially]
* Can not multiplex, even if we wanted to!
* Re-using existing connection! (#0) with host daac.ornl.gov
* Connected to daac.ornl.gov (160.91.19.24) port 443 (#0)
> GET /daacdata/doesntmater?code=XXX&state=XXX HTTP/1.1
> Host: daac.ornl.gov
> User-Agent: curl/7.74.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 302 Found
< Date: Mon, 25 Apr 2022 13:03:08 GMT
< Server: Apache
< Strict-Transport-Security: max-age=31536000
* Added cookie AppAuth="XXX" for domain daac.ornl.gov, path /, expire 0
< Set-Cookie: AppAuth=XXX; Path=/
< Location: https://daac.ornl.gov/daacdata/daymet/Daymet_V4_Stn_Level_CrossVal/data/stnxvalmeta_tmin_pr_2016.txt
< Content-Length: 284
< Content-Type: text/html; charset=iso-8859-1
<
* Ignoring the response-body
* Connection #0 to host daac.ornl.gov left intact
* Issue another request to this URL: 'https://daac.ornl.gov/daacdata/daymet/Daymet_V4_Stn_Level_CrossVal/data/stnxvalmeta_tmin_pr_2016.txt'
* Couldn't find host daac.ornl.gov in the .netrc file; using defaults
* Found bundle for host daac.ornl.gov: XXX [serially]
* Can not multiplex, even if we wanted to!
* Connection 0 seems to be dead!
* Closing connection 0
* TLSv1.2 (OUT), TLS alert, close notify (256):
* Hostname daac.ornl.gov was found in DNS cache
* Trying 160.91.19.24:443...
* Connected to daac.ornl.gov (160.91.19.24) port 443 (#2)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
* CApath: /etc/ssl/certs
* SSL re-using session ID
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
* subject: C=US; ST=Tennessee; L=Oak Ridge; O=Oak Ridge National Laboratory; OU=DAAC; CN=*.ornl.gov
* start date: Jul 14 00:00:00 2020 GMT
* expire date: Oct 11 12:00:00 2022 GMT
* subjectAltName: host "daac.ornl.gov" matched cert's "*.ornl.gov"
* issuer: C=US; O=DigiCert Inc; CN=DigiCert SHA2 Secure Server CA
* SSL certificate verify ok.
> GET /daacdata/daymet/Daymet_V4_Stn_Level_CrossVal/data/stnxvalmeta_tmin_pr_2016.txt HTTP/1.1
> Host: daac.ornl.gov
> User-Agent: curl/7.74.0
> Accept: */*
> Cookie: AppAuth=XXX
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Date: Mon, 25 Apr 2022 13:03:09 GMT
< Server: Apache
< Strict-Transport-Security: max-age=31536000
< Last-Modified: Mon, 07 Dec 2020 22:32:53 GMT
< ETag: "5b5-5b5e76833e87e"
< Accept-Ranges: bytes
< Content-Length: 1461
< Vary: Accept-Encoding
< Content-Type: text/plain; charset=UTF-8
<
station_id,station_name,stn_lat,stn_lon,stnz
RQC00660152,AGUIRRE,17.9556,-66.2222,7.6
RQC00664193,GUAYAMA 2E,17.9783,-66.0875,21.9
RQC00665693,MAGUEYES IS,17.9722,-67.0461,3.7
VQC00671740,CHRISTIANSTED FT,17.7469,-64.7014,9.1
VQW00011624,CHRISTIANSTED AP,17.7028,-64.8056,18.6
RQC00660061,ADJUNTAS SUBSTN,18.1747,-66.7978,557.8
RQC00660158,AIBONITO 1 S,18.1281,-66.2642,710.2
RQC00660426,ARECIBO OBSY,18.3494,-66.7525,323.1
RQC00662801,COLOSO,18.3808,-67.1569,12.2
RQC00663431,DOS BOCAS,18.3361,-66.6667,61.0
RQC00664614,HUMACAO NATURAL RESERVE,18.1506,-66.7719,2.7
RQC00664702,ISABELA SUBSTN,18.4653,-67.0525,128.0
RQC00665097,LAJAS SUBSTN,18.0331,-67.0722,27.4
RQC00665807,MANATI 2 E,18.4308,-66.4661,76.2
RQC00665908,MARICAO 2 SSW,18.1511,-66.9889,863.2
RQC00666730,PALMAREJO VEGA BAJA,18.385,-66.43,170.7
RQC00667292,PONCE 4 E,18.0258,-66.5253,21.3
RQC00668306,RIO PIEDRAS EXP STN,18.3906,-66.0542,28.0
RQC00669415,TOA BAJA LEVITTOWN,18.4356,-66.1678,8.5
RQC00669432,TORO NEGRO FOREST,18.1731,-66.4928,868.1
RQC00669521,TRUJILLO ALTO 2 SSW,18.3283,-66.0164,35.1
RQW00011641,SAN JUAN L M MARIN AP,18.4325,-66.0108,2.7
RQC00665064,JUNCOS 1 SE,18.2264,-65.9114,64.9
RQC00666343,CULEBRA HILL,18.2972,-65.29,71.3
RQC00666725,PALMA SOLA,18.3169,-65.8664,466.0
RQC00668814,WFO SAN JUAN,18.4311,-65.9917,3.0
RQW00011630,ROOSEVELT ROADS,18.255,-65.6408,10.1
VQC00677600,REDHOOK BAY ST THOMAS,18.3236,-64.8542,0.6
VQW00011640,CHARLOTTE AMALIE AP,18.3331,-64.9667,6.1
* Connection #2 to host daac.ornl.gov left intact
The --user version works if -L is replaced with --location-trusted and --anyauth is added.
in your netrc you specifically say "this is the username:password for urs.earthdata.nasa.gov", and your command first connects to a different domain: daac.ornl.gov. so your netrc does not trigger on daac.ornl.gov, your "broken" curl command sends the username:password to daac.ornl.gov and and daac.ornl.gov does not like this invalid username:password and your curl request get rejected. meanwhile your working curl command does not send username:password to daac.ornl.gov, then daac.ornl.gov proceed to redirect you to to urs.earthdata.nasa.gov , here your netrc kicks in and your username:password gets sent to urs.earthdata.nasa.gov as it should..
unfortunately curl does not (currently?) support loading netrc from stdin, and does not support setting username:password for specific domains via arguments (not as far as i know anyway?), so your best bet is probably:
NETRCTMPFILE="$(mktemp)"
echo 'machine urs.earthdata.nasa.gov login USERNAME password PASSWORD' > "$NETRCTMPFILE";
curl -b ~/.cookies -c ~/.cookies -L --netrc-file "$NETRCTMPFILE" https://daac.ornl.gov/daacdata/daymet/Daymet_V4_Stn_Level_CrossVal/data/stnxvalmeta_tmin_pr_2016.txt
rm "$NETRCTMPFILE"
You're using quotes around the username and password. In my experience this causes issues.
try
curl -b ~/.cookies -c ~/.cookies -L --user USERNAME:PASSWORD https://daac.ornl.gov/daacdata/daymet/Daymet_V4_Stn_Level_CrossVal/data/stnxvalmeta_tmin_pr_2016.txt
This has issues with your user and password being present in bash history.
This link has a lot of good info on other ways to do this safely too
https://stackoverflow.com/a/56130884/496405
I am trying to send a simple curl request:
curl -k -i --key ./key.pem --cert ./cert.pem https://target_ip/whatever/
The problem I'm having is that it does not send any certificate. The validation clearly passes as otherwise I was getting errors such as the key does not match but then I can see in wireshark that the certificates are not being sent in the TCP connection around Client Hello. Switches like --verbose or --cacert don't do much either.
I am able to send the very same certificates through postman successfully.
I have tried sending the same curl request from various sources such as my WSL2 ubuntu, a debian container in the cloud, a VM, ...
Any tips why it is not sending the certs?
EDIT I - output from curl -v
* Trying 52.xxx.xxx.xx:443...
* TCP_NODELAY set
* Connected to 52.xxx.xxx.xx (52.xxx.xxx.xx) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
* subject: CN=NGINXIngressController
* start date: Aug 10 18:08:13 2020 GMT
* expire date: Aug 10 18:08:13 2021 GMT
* issuer: CN=NGINXIngressController
* SSL certificate verify result: self signed certificate (18), continuing anyway.
> GET /whatever/ HTTP/1.1
> Host: custom.localhost.dev
> User-Agent: curl/7.68.0
> Accept: */*
> Authorization: Bearer eyJ0...
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 403 Forbidden
HTTP/1.1 403 Forbidden
< Server: nginx/1.19.0
Server: nginx/1.19.0
< Date: Mon, 10 Aug 2020 22:23:24 GMT
Date: Mon, 10 Aug 2020 22:23:24 GMT
< Content-Type: text/html
Content-Type: text/html
< Content-Length: 153
Content-Length: 153
< Connection: keep-alive
Connection: keep-alive
<
<html>
<head><title>403 Forbidden</title></head>
<body>
<center><h1>403 Forbidden</h1></center>
<hr><center>nginx/1.19.0</center>
</body>
</html>
* Connection #0 to host 52.xxx.xxx.xx left intact
EDIT II - wireshark captures
It seems to be too much of a hassle to anonymise pcap, so here's just some snaps. Hopefully you'll be able to see all you need. I have highlighted the packet where I do (not) see the cert being sent. Note that I am running the postman on my windows workstation, whereas the curl is in the WSL2, hence the different source addresses. Other hosts for curl did behave the same though.
Curl
Postman
EDIT III - Client Hellos
Curl
Postman
The ClientHello shows a clear difference: postman uses the server_name extension (SNI) to provide the expected hostname while curl does not.
This likely triggers a different part of the configuration in the web server: postman triggers access to the specific virtual host given as server_name while curl will probably run into the default configuration. Assuming that only the specific virtual host enables client certificates this explains why the CertificateRequest is send by the server only to postman but not to curl.
It is unclear what this hostname is, but based on the length it cannot be an IP address. Thus postman somehow must know the expected hostname of the server even though it is claimed that the access was done with https://target_ip/ only, i.e. without a given hostname. curl cannot derive from this URL the expected hostname and thus cannot set server_name. To make curl be aware of the hostname to set server_name while still being able to access a specific IP use the --resolve option:
curl --resolve hostname:443:target_ip https://hostname/
Update: I just run the command inside the docker of the same Linux machine, and it worked. Therefore might be an issue related to Linux distro. I personally suspect something related to SSL certifications.
I set up a Kubernetes cluster in AWS EKS and a whole running environment by using MacBook. However, I found out myself cannot setup kubectl correctly in my Linux machine (ArchLinux).
I've tried to run the kubectl --v=1000 get svc (some cluster info was masked)
I1020 11:01:44.053581 3266 loader.go:359] Config loaded from file /home/realturner/.kube/config
I1020 11:01:44.054963 3266 round_trippers.go:419] curl -k -v -XGET -H "Accept: application/json, */*" -H "User-Agent: kubectl/v1.14.7 (linux/amd64) kubernetes/1861c59" 'https://XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.eks.amazonaws.com/api?timeout=32s'
I1020 11:01:44.299305 3266 round_trippers.go:438] GET https://XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.eks.amazonaws.com/api?timeout=32s in 244 milliseconds
I1020 11:01:44.299331 3266 round_trippers.go:444] Response Headers:
I1020 11:01:44.299367 3266 cached_discovery.go:121] skipped caching discovery info due to Get https://XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.eks.amazonaws.com/api?timeout=32s: dial tcp: lookup XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.eks.amazonaws.com on [::1]:53: read udp [::1]:34122->[::1]:53: read: connection refused
When compared to another machine, the successful one replies headers and body
I1020 11:03:44.358266 1675 loader.go:359] Config loaded from file /Users/realturner/.kube/config-tv
I1020 11:03:44.359417 1675 round_trippers.go:419] curl -k -v -XGET -H "Accept: application/json, */*" -H "User-Agent: kubectl/v1.14.6 (darwin/amd64) kubernetes/96fac5c" 'https://XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.eks.amazonaws.com/api?timeout=32s'
I1020 11:03:46.186432 1675 round_trippers.go:438] GET https://XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.eks.amazonaws.com/api?timeout=32s 200 OK in 1826 milliseconds
I1020 11:03:46.186481 1675 round_trippers.go:444] Response Headers:
I1020 11:03:46.186498 1675 round_trippers.go:447] Content-Length: 149
I1020 11:03:46.186512 1675 round_trippers.go:447] Date: Sun, 20 Oct 2019 03:03:46 GMT
I1020 11:03:46.186525 1675 round_trippers.go:447] Audit-Id: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
I1020 11:03:46.186538 1675 round_trippers.go:447] Content-Type: application/json
I1020 11:03:46.262841 1675 request.go:942] Response Body: {"kind":"APIVersions","versions":["v1"],"serverAddressByClientCIDRs":[{"clientCIDR":"0.0.0.0/0","serverAddress":"ip-10-xxx-xxx-xxx.ec2.internal:443"}]}
I'd suspect a network or firewall problem, but simply doing curl to that endpoint do have some response, though lack of permission:
$ curl -k -v -XGET -H "Accept: application/json, */*" -H "User-Agent: kubectl/v1.14.7 (linux/amd64) kubernetes/1861c59" 'https://XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.eks.amazonaws.com/api?timeout=32s'
Note: Unnecessary use of -X or --request, GET is already inferred.
* Trying x.xxx.xxx.xx:443...
* TCP_NODELAY set
* Connected to XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.eks.amazonaws.com (x.xxx.xxx.xx) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Request CERT (13):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
* subject: CN=kube-apiserver
* start date: Oct 17 10:29:43 2019 GMT
* expire date: Oct 16 10:29:44 2020 GMT
* issuer: CN=kubernetes
* SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x55cb670e87b0)
> GET /api?timeout=32s HTTP/2
> Host: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.eks.amazonaws.com
> Accept: application/json, */*
> User-Agent: kubectl/v1.14.7 (linux/amd64) kubernetes/1861c59
>
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
< HTTP/2 403
< audit-id: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
< content-type: application/json
< x-content-type-options: nosniff
< content-length: 188
< date: Sun, 20 Oct 2019 14:56:42 GMT
<
{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"forbidden: User \"system:anonymous\" cannot get path \"/api\"","reason":"Forbidden","details":{},"code":403}
* Connection #0 to host XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.eks.amazonaws.com left intact
Edit - here's my .kube/config, same as .kube/tv-config (some items masked). It's generated by aws eks update-kubeconfig --name <Cluster>
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: <Same as the one in `Certificate authority
` of EKS>
server: https://<PREFIX>.<REGION>.eks.amazonaws.com
name: arn:aws:eks:<REGION>:<AccountId>:cluster/<Cluster>
contexts:
- context:
cluster: arn:aws:eks:<REGION>:<AccountId>:cluster/<Cluster>
user: arn:aws:eks:<REGION>:<AccountId>:cluster/<Cluster>
name: arn:aws:eks:<REGION>:<AccountId>:cluster/<Cluster>
current-context: arn:aws:eks:<REGION>:<AccountId>:cluster/<Cluster>
kind: Config
preferences: {}
users:
- name: arn:aws:eks:<REGION>:<AccountId>:cluster/<Cluster>
user:
exec:
apiVersion: client.authentication.k8s.io/v1alpha1
args:
- --region
- <REGION>
- eks
- get-token
- --cluster-name
- <Cluster>
command: aws
Wow, it turns out to be the DNS resolution problem, despite that I used the newly install the system for several days without noticing it.
Previously I just tried getent hosts <DNS> for DNS resolution and use curl -v <PREFIX>.<REGION>.eks.amazonaws.com> for DNS resolution test. They both had replied correctly before I found my /etc/resolv.conf is actually empty.
I have missed configuring systemd's DNS resolution. As documented here:
Note that if you want to take advantage of automatic DNS configuration from DHCP, you need to enable systemd-resolved and symlink /run/systemd/resolve/resolv.conf to /etc/resolv.conf
After symbolic linking now it just works as expected!
I'm trying to follow the xively cURL tutorial. I have created new device in develop mode, copied the default auto-generated API key (with READ, WRITE, CREATE, DELETE permissions) to the example in tutorial and get response:
{"title":"Forbidden","errors":"You do not have the necessary permissions to access this resource"}
I must be missing some obvious step. Do I need to activate the API key somehow before using it in scripts?
The cURL command:
curl --request POST \
--data '{"title":"My feed", "version":"1.0.0"}' \
--header "X-ApiKey: cPHLfGw1WJdMAbU8FzbfsdFyJ8suayHEH3OChRrkpYwQCmrb" \
--verbose \
https://api.xively.com/v2/feeds
full verbose output:
* About to connect() to api.xively.com port 443 (#0)
* Trying 216.52.233.120...
* Connected to api.xively.com (216.52.233.120) port 443 (#0)
* successfully set certificate verify locations:
* CAfile: none
CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Server key exchange (12):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using ECDHE-RSA-RC4-SHA
* Server certificate:
* subject: C=US; postalCode=01801; ST=MA; L=Woburn; street=First Floor; street=500 Unicorn Park Drive; O=LogMeIn Inc.; OU=Secure Link SSL Wildcard; CN=*.xively.com
* start date: 2013-05-07 00:00:00 GMT
* expire date: 2014-04-27 23:59:59 GMT
* subjectAltName: api.xively.com matched
* issuer: C=US; O=Network Solutions L.L.C.; CN=Network Solutions Certificate Authority
* SSL certificate verify ok.
> POST /v2/feeds HTTP/1.1
> User-Agent: curl/7.29.0
> Host: api.xively.com
> Accept: */*
> X-ApiKey: cPHLfGw1WJdMAbU8FzbfsdFyJ8suayHEH3OChRrkpYwQCmrb
> Content-Length: 38
> Content-Type: application/x-www-form-urlencoded
>
* upload completely sent off: 38 out of 38 bytes
< HTTP/1.1 403 Forbidden
< Date: Sat, 30 Nov 2013 11:03:15 GMT
< Content-Type: application/json; charset=utf-8
< Content-Length: 98
< Connection: keep-alive
< X-Request-Id: 6cbb9676b448a4967187271dd246b423f7da2e39
<
* Connection #0 to host api.xively.com left intact
{"title":"Forbidden","errors":"You do not have the necessary permissions to access this resource"}
It depends what you are trying to do. POST requests to api.xively.com/v2/feeds are no longer supported with any API key. This is because, since the implementation of devices, programatic creation of feeds directly is no longer supported.
This is an oversight in the Xively tutorial and I will inform the appropriate people to make sure that it gets changed.
In the mean time, since you have already created a device, you are basically ready to start the cURL tutorial at step 3 "Update a Feed". Use the API key and Feed ID from the development device you have already created on the website. Make sure to change your URL, body, and from a POST request to a PUT request.
I think you need to add your feed id onto the end of the URL.
I am having trouble creating a fulfillment via the API. I am currently using curl to test, but am receiving a "Required parameter missing" error.
Anybody able to catch the error?
curl command:
curl -i -v -H "Accept: application/json" -X POST -d "{'fulfillment': {'tracking_number': null,'line_items': [{'id': 322959149}]}}" https://[apikey]:[password]#[shop-name].myshopify.com/admin/orders/186122043/fulfillments.json
curl output:
* About to connect() to [shop-name].myshopify.com port 443 (#0)
* Trying 204.93.213.94...
* connected
* Connected to [shop-name].myshopify.com (204.93.213.94) port 443 (#0)
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Server key exchange (12):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using DHE-RSA-AES256-SHA
* Server certificate:
* subject: serialNumber=inVwoUGzj3duFEUBdzDQrbS2kO1qx0Vr; C=CA; ST=Ontario; L=Ottawa; O=Jaded Pixel Technologies Inc.; CN=*.myshopify.com
* start date: 2010-05-10 22:23:08 GMT
* expire date: 2015-08-12 19:17:14 GMT
* subjectAltName: [shop-name].myshopify.com matched
* issuer: C=US; O=Equifax; OU=Equifax Secure Certificate Authority
* SSL certificate verify ok.
* Server auth using Basic with user '4a93b24da162b9ce3f6426206b63e692'
> POST /admin/orders/186122043/fulfillments.json HTTP/1.1
> Authorization: Basic NGE5M2IyNGRhMTYyYjljZTNmNjQyNjIwNmI2M2U2OTI6ZDljMWYwMDNlZWZiMDE1MmQ1MzExMGNjMzk1N2MzMDI=
> User-Agent: curl/7.24.0 (x86_64-apple-darwin12.0) libcurl/7.24.0 OpenSSL/0.9.8y zlib/1.2.5
> Host: [shop-name].myshopify.com
> Accept: application/json
> Content-Length: 76
> Content-Type: application/x-www-form-urlencoded
>
* upload completely sent off: 76 out of 76 bytes
< HTTP/1.1 400 Bad Request
HTTP/1.1 400 Bad Request
< Server: nginx
Server: nginx
< Date: Mon, 07 Oct 2013 21:10:02 GMT
Date: Mon, 07 Oct 2013 21:10:02 GMT
< Content-Type: application/json; charset=utf-8
Content-Type: application/json; charset=utf-8
< Transfer-Encoding: chunked
Transfer-Encoding: chunked
< Status: 400 Bad Request
Status: 400 Bad Request
< X-ShopId: 2235177
X-ShopId: 2235177
< X-Shopify-Asset-Version: 529b6843243276ab3e385c6b38a3bd7b17b4b92f
X-Shopify-Asset-Version: 529b6843243276ab3e385c6b38a3bd7b17b4b92f
< X-Shopify-Shop-Api-Call-Limit: 2/500
X-Shopify-Shop-Api-Call-Limit: 2/500
< HTTP_X_SHOPIFY_SHOP_API_CALL_LIMIT: 2/500
HTTP_X_SHOPIFY_SHOP_API_CALL_LIMIT: 2/500
< X-Frame-Options: DENY
X-Frame-Options: DENY
< X-UA-Compatible: IE=Edge,chrome=1
X-UA-Compatible: IE=Edge,chrome=1
< Set-Cookie: _secure_session_id=989c50f70ca8fd46d381770aa5ad3e72; path=/; secure; HttpOnly
Set-Cookie: _secure_session_id=989c50f70ca8fd46d381770aa5ad3e72; path=/; secure; HttpOnly
< X-Request-Id: 3c394e98aebf3ae3ba4c3f75b8f069be
X-Request-Id: 3c394e98aebf3ae3ba4c3f75b8f069be
<
* Connection #0 to host [shop-name].myshopify.com left intact
{"errors":{"fulfillment":"Required parameter missing"}}
* Closing connection #0
* SSLv3, TLS alert, Client hello (1):
Your ' should "
curl -i -v -H "Accept: application/json" -X POST -d '{"fulfillment": {"tracking_number": null,"line_items": [{"id": 322959149}]}}' https://[apikey]:[password]#[shop-name].myshopify.com/admin/orders/186122043/fulfillments.json
following link should help you.