With curl 7.74.0, the command
$ curl -b ~/.cookies -c ~/.cookies -L --user 'USERNAME:PASSWORD' https://daac.ornl.gov/daacdata/daymet/Daymet_V4_Stn_Level_CrossVal/data/stnxvalmeta_tmin_pr_2016.txt
yields HTTP 401 (verbose output listed below). However, the following works:
$ echo 'machine urs.earthdata.nasa.gov login USERNAME password PASSWORD' >>~/.netrc
$ curl -b ~/.cookies -c ~/.cookies -L --netrc https://daac.ornl.gov/daacdata/daymet/Daymet_V4_Stn_Level_CrossVal/data/stnxvalmeta_tmin_pr_2016.txt
Why is that? Shouldn't these approaches be equivalent? Perhaps there's an extra option I need to convince curl to pass the credentials set with --user down to urs.earthdata.nasa.gov.
You can create an Earthdata account at https://urs.earthdata.nasa.gov/users/new to test these commands.
(N.B. The --user version works if you already have a valid login cookie from the --netrc version. The question is why login doesn't work.)
The -v output for the --user version is (authenticating cookies etc. censored with "XXX"):
* Trying 160.91.19.24:443...
* Connected to daac.ornl.gov (160.91.19.24) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
* CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
* subject: C=US; ST=Tennessee; L=Oak Ridge; O=Oak Ridge National Laboratory; OU=DAAC; CN=*.ornl.gov
* start date: Jul 14 00:00:00 2020 GMT
* expire date: Oct 11 12:00:00 2022 GMT
* subjectAltName: host "daac.ornl.gov" matched cert's "*.ornl.gov"
* issuer: C=US; O=DigiCert Inc; CN=DigiCert SHA2 Secure Server CA
* SSL certificate verify ok.
* Server auth using Basic with user 'Kodiologist'
> GET /daacdata/daymet/Daymet_V4_Stn_Level_CrossVal/data/stnxvalmeta_tmin_pr_2016.txt HTTP/1.1
> Host: daac.ornl.gov
> Authorization: Basic XXX
> User-Agent: curl/7.74.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 401 Unauthorized
< Date: Mon, 25 Apr 2022 13:02:02 GMT
< Server: Apache
< Strict-Transport-Security: max-age=31536000
< Content-Length: 381
< Content-Type: text/html; charset=iso-8859-1
<
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>401 Unauthorized</title>
</head><body>
<h1>Unauthorized</h1>
<p>This server could not verify that you
are authorized to access the document
requested. Either you supplied the wrong
credentials (e.g., bad password), or your
browser doesn't understand how to supply
the credentials required.</p>
</body></html>
* Connection #0 to host daac.ornl.gov left intact
The -v output for the --netrc version is:
* Couldn't find host daac.ornl.gov in the .netrc file; using defaults
* Trying 160.91.19.24:443...
* Connected to daac.ornl.gov (160.91.19.24) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
* CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
* subject: C=US; ST=Tennessee; L=Oak Ridge; O=Oak Ridge National Laboratory; OU=DAAC; CN=*.ornl.gov
* start date: Jul 14 00:00:00 2020 GMT
* expire date: Oct 11 12:00:00 2022 GMT
* subjectAltName: host "daac.ornl.gov" matched cert's "*.ornl.gov"
* issuer: C=US; O=DigiCert Inc; CN=DigiCert SHA2 Secure Server CA
* SSL certificate verify ok.
> GET /daacdata/daymet/Daymet_V4_Stn_Level_CrossVal/data/stnxvalmeta_tmin_pr_2016.txt HTTP/1.1
> Host: daac.ornl.gov
> User-Agent: curl/7.74.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 302 Found
< Date: Mon, 25 Apr 2022 13:03:08 GMT
< Server: Apache
< Strict-Transport-Security: max-age=31536000
< Location: https://urs.earthdata.nasa.gov/oauth/authorize?app_type=401&client_id=XXXA&response_type=code&redirect_uri=https%3A%2F%2Fdaac.ornl.gov%2Fdaacdata%2Fdoesntmater&state=XXX
< Content-Length: 518
< Content-Type: text/html; charset=iso-8859-1
<
* Ignoring the response-body
* Connection #0 to host daac.ornl.gov left intact
* Issue another request to this URL: 'https://urs.earthdata.nasa.gov/oauth/authorize?app_type=401&client_id=XXX&response_type=code&redirect_uri=https%3A%2F%2Fdaac.ornl.gov%2Fdaacdata%2Fdoesntmater&state=XXX'
* Trying 198.118.243.33:443...
* Connected to urs.earthdata.nasa.gov (198.118.243.33) port 443 (#1)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
* CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
* subject: C=US; ST=Maryland; L=Greenbelt; jurisdictionC=US; O=NASA (National Aeronautics and Space Administration); businessCategory=Government Entity; serialNumber=1958-07-29; CN=urs.earthdata.nasa.gov
* start date: Aug 27 16:09:18 2021 GMT
* expire date: Sep 12 16:09:18 2022 GMT
* subjectAltName: host "urs.earthdata.nasa.gov" matched cert's "urs.earthdata.nasa.gov"
* issuer: C=US; O=Entrust, Inc.; OU=See www.entrust.net/legal-terms; OU=(c) 2014 Entrust, Inc. - for authorized use only; CN=Entrust Certification Authority - L1M
* SSL certificate verify ok.
* Server auth using Basic with user 'Kodiologist'
> GET /oauth/authorize?app_type=401&client_id=XXX&response_type=code&redirect_uri=https%3A%2F%2Fdaac.ornl.gov%2Fdaacdata%2Fdoesntmater&state=XXX HTTP/1.1
> Host: urs.earthdata.nasa.gov
> Authorization: Basic XXX
> User-Agent: curl/7.74.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 302 Found
< Server: nginx/1.20.1
< Date: Mon, 25 Apr 2022 13:03:08 GMT
< Content-Type: text/html; charset=utf-8
< Transfer-Encoding: chunked
< Connection: keep-alive
< X-Frame-Options: SAMEORIGIN
< X-XSS-Protection: 1; mode=block
< X-Content-Type-Options: nosniff
< X-Download-Options: noopen
< X-Permitted-Cross-Domain-Policies: none
< Referrer-Policy: strict-origin-when-cross-origin
< Cache-Control: no-store
< Pragma: no-cache
< Expires: Fri, 01 Jan 1990 00:00:00 GMT
< Access-Control-Allow-Origin: null
< Access-Control-Allow-Credentials: true
< Access-Control-Allow-Methods: GET, POST
< Access-Control-Expose-Headers: true
< Location: https://daac.ornl.gov/daacdata/doesntmater?code=XXX&state=XXX
* Added cookie urs_user_already_logged="yes" for domain earthdata.nasa.gov, path /, expire 1650978188
< Set-Cookie: urs_user_already_logged=yes; domain=earthdata.nasa.gov; path=/; expires=Tue, 26 Apr 2022 13:03:08 GMT; secure; HttpOnly
* Added cookie _urs-gui_session="XXX" for domain urs.earthdata.nasa.gov, path /, expire 1650978188
< Set-Cookie: _urs-gui_session=XXX; path=/; expires=Tue, 26 Apr 2022 13:03:08 GMT; HttpOnly
< X-Request-Id: XXX
< X-Runtime: 0.053834
< Strict-Transport-Security: max-age=31536000
<
* Ignoring the response-body
* Connection #1 to host urs.earthdata.nasa.gov left intact
* Issue another request to this URL: 'https://daac.ornl.gov/daacdata/doesntmater?code=XXX&state=XXX'
* Couldn't find host daac.ornl.gov in the .netrc file; using defaults
* Found bundle for host daac.ornl.gov: XXX [serially]
* Can not multiplex, even if we wanted to!
* Re-using existing connection! (#0) with host daac.ornl.gov
* Connected to daac.ornl.gov (160.91.19.24) port 443 (#0)
> GET /daacdata/doesntmater?code=XXX&state=XXX HTTP/1.1
> Host: daac.ornl.gov
> User-Agent: curl/7.74.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 302 Found
< Date: Mon, 25 Apr 2022 13:03:08 GMT
< Server: Apache
< Strict-Transport-Security: max-age=31536000
* Added cookie AppAuth="XXX" for domain daac.ornl.gov, path /, expire 0
< Set-Cookie: AppAuth=XXX; Path=/
< Location: https://daac.ornl.gov/daacdata/daymet/Daymet_V4_Stn_Level_CrossVal/data/stnxvalmeta_tmin_pr_2016.txt
< Content-Length: 284
< Content-Type: text/html; charset=iso-8859-1
<
* Ignoring the response-body
* Connection #0 to host daac.ornl.gov left intact
* Issue another request to this URL: 'https://daac.ornl.gov/daacdata/daymet/Daymet_V4_Stn_Level_CrossVal/data/stnxvalmeta_tmin_pr_2016.txt'
* Couldn't find host daac.ornl.gov in the .netrc file; using defaults
* Found bundle for host daac.ornl.gov: XXX [serially]
* Can not multiplex, even if we wanted to!
* Connection 0 seems to be dead!
* Closing connection 0
* TLSv1.2 (OUT), TLS alert, close notify (256):
* Hostname daac.ornl.gov was found in DNS cache
* Trying 160.91.19.24:443...
* Connected to daac.ornl.gov (160.91.19.24) port 443 (#2)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
* CApath: /etc/ssl/certs
* SSL re-using session ID
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
* subject: C=US; ST=Tennessee; L=Oak Ridge; O=Oak Ridge National Laboratory; OU=DAAC; CN=*.ornl.gov
* start date: Jul 14 00:00:00 2020 GMT
* expire date: Oct 11 12:00:00 2022 GMT
* subjectAltName: host "daac.ornl.gov" matched cert's "*.ornl.gov"
* issuer: C=US; O=DigiCert Inc; CN=DigiCert SHA2 Secure Server CA
* SSL certificate verify ok.
> GET /daacdata/daymet/Daymet_V4_Stn_Level_CrossVal/data/stnxvalmeta_tmin_pr_2016.txt HTTP/1.1
> Host: daac.ornl.gov
> User-Agent: curl/7.74.0
> Accept: */*
> Cookie: AppAuth=XXX
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Date: Mon, 25 Apr 2022 13:03:09 GMT
< Server: Apache
< Strict-Transport-Security: max-age=31536000
< Last-Modified: Mon, 07 Dec 2020 22:32:53 GMT
< ETag: "5b5-5b5e76833e87e"
< Accept-Ranges: bytes
< Content-Length: 1461
< Vary: Accept-Encoding
< Content-Type: text/plain; charset=UTF-8
<
station_id,station_name,stn_lat,stn_lon,stnz
RQC00660152,AGUIRRE,17.9556,-66.2222,7.6
RQC00664193,GUAYAMA 2E,17.9783,-66.0875,21.9
RQC00665693,MAGUEYES IS,17.9722,-67.0461,3.7
VQC00671740,CHRISTIANSTED FT,17.7469,-64.7014,9.1
VQW00011624,CHRISTIANSTED AP,17.7028,-64.8056,18.6
RQC00660061,ADJUNTAS SUBSTN,18.1747,-66.7978,557.8
RQC00660158,AIBONITO 1 S,18.1281,-66.2642,710.2
RQC00660426,ARECIBO OBSY,18.3494,-66.7525,323.1
RQC00662801,COLOSO,18.3808,-67.1569,12.2
RQC00663431,DOS BOCAS,18.3361,-66.6667,61.0
RQC00664614,HUMACAO NATURAL RESERVE,18.1506,-66.7719,2.7
RQC00664702,ISABELA SUBSTN,18.4653,-67.0525,128.0
RQC00665097,LAJAS SUBSTN,18.0331,-67.0722,27.4
RQC00665807,MANATI 2 E,18.4308,-66.4661,76.2
RQC00665908,MARICAO 2 SSW,18.1511,-66.9889,863.2
RQC00666730,PALMAREJO VEGA BAJA,18.385,-66.43,170.7
RQC00667292,PONCE 4 E,18.0258,-66.5253,21.3
RQC00668306,RIO PIEDRAS EXP STN,18.3906,-66.0542,28.0
RQC00669415,TOA BAJA LEVITTOWN,18.4356,-66.1678,8.5
RQC00669432,TORO NEGRO FOREST,18.1731,-66.4928,868.1
RQC00669521,TRUJILLO ALTO 2 SSW,18.3283,-66.0164,35.1
RQW00011641,SAN JUAN L M MARIN AP,18.4325,-66.0108,2.7
RQC00665064,JUNCOS 1 SE,18.2264,-65.9114,64.9
RQC00666343,CULEBRA HILL,18.2972,-65.29,71.3
RQC00666725,PALMA SOLA,18.3169,-65.8664,466.0
RQC00668814,WFO SAN JUAN,18.4311,-65.9917,3.0
RQW00011630,ROOSEVELT ROADS,18.255,-65.6408,10.1
VQC00677600,REDHOOK BAY ST THOMAS,18.3236,-64.8542,0.6
VQW00011640,CHARLOTTE AMALIE AP,18.3331,-64.9667,6.1
* Connection #2 to host daac.ornl.gov left intact
The --user version works if -L is replaced with --location-trusted and --anyauth is added.
in your netrc you specifically say "this is the username:password for urs.earthdata.nasa.gov", and your command first connects to a different domain: daac.ornl.gov. so your netrc does not trigger on daac.ornl.gov, your "broken" curl command sends the username:password to daac.ornl.gov and and daac.ornl.gov does not like this invalid username:password and your curl request get rejected. meanwhile your working curl command does not send username:password to daac.ornl.gov, then daac.ornl.gov proceed to redirect you to to urs.earthdata.nasa.gov , here your netrc kicks in and your username:password gets sent to urs.earthdata.nasa.gov as it should..
unfortunately curl does not (currently?) support loading netrc from stdin, and does not support setting username:password for specific domains via arguments (not as far as i know anyway?), so your best bet is probably:
NETRCTMPFILE="$(mktemp)"
echo 'machine urs.earthdata.nasa.gov login USERNAME password PASSWORD' > "$NETRCTMPFILE";
curl -b ~/.cookies -c ~/.cookies -L --netrc-file "$NETRCTMPFILE" https://daac.ornl.gov/daacdata/daymet/Daymet_V4_Stn_Level_CrossVal/data/stnxvalmeta_tmin_pr_2016.txt
rm "$NETRCTMPFILE"
You're using quotes around the username and password. In my experience this causes issues.
try
curl -b ~/.cookies -c ~/.cookies -L --user USERNAME:PASSWORD https://daac.ornl.gov/daacdata/daymet/Daymet_V4_Stn_Level_CrossVal/data/stnxvalmeta_tmin_pr_2016.txt
This has issues with your user and password being present in bash history.
This link has a lot of good info on other ways to do this safely too
https://stackoverflow.com/a/56130884/496405
Update: I just run the command inside the docker of the same Linux machine, and it worked. Therefore might be an issue related to Linux distro. I personally suspect something related to SSL certifications.
I set up a Kubernetes cluster in AWS EKS and a whole running environment by using MacBook. However, I found out myself cannot setup kubectl correctly in my Linux machine (ArchLinux).
I've tried to run the kubectl --v=1000 get svc (some cluster info was masked)
I1020 11:01:44.053581 3266 loader.go:359] Config loaded from file /home/realturner/.kube/config
I1020 11:01:44.054963 3266 round_trippers.go:419] curl -k -v -XGET -H "Accept: application/json, */*" -H "User-Agent: kubectl/v1.14.7 (linux/amd64) kubernetes/1861c59" 'https://XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.eks.amazonaws.com/api?timeout=32s'
I1020 11:01:44.299305 3266 round_trippers.go:438] GET https://XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.eks.amazonaws.com/api?timeout=32s in 244 milliseconds
I1020 11:01:44.299331 3266 round_trippers.go:444] Response Headers:
I1020 11:01:44.299367 3266 cached_discovery.go:121] skipped caching discovery info due to Get https://XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.eks.amazonaws.com/api?timeout=32s: dial tcp: lookup XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.eks.amazonaws.com on [::1]:53: read udp [::1]:34122->[::1]:53: read: connection refused
When compared to another machine, the successful one replies headers and body
I1020 11:03:44.358266 1675 loader.go:359] Config loaded from file /Users/realturner/.kube/config-tv
I1020 11:03:44.359417 1675 round_trippers.go:419] curl -k -v -XGET -H "Accept: application/json, */*" -H "User-Agent: kubectl/v1.14.6 (darwin/amd64) kubernetes/96fac5c" 'https://XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.eks.amazonaws.com/api?timeout=32s'
I1020 11:03:46.186432 1675 round_trippers.go:438] GET https://XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.eks.amazonaws.com/api?timeout=32s 200 OK in 1826 milliseconds
I1020 11:03:46.186481 1675 round_trippers.go:444] Response Headers:
I1020 11:03:46.186498 1675 round_trippers.go:447] Content-Length: 149
I1020 11:03:46.186512 1675 round_trippers.go:447] Date: Sun, 20 Oct 2019 03:03:46 GMT
I1020 11:03:46.186525 1675 round_trippers.go:447] Audit-Id: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
I1020 11:03:46.186538 1675 round_trippers.go:447] Content-Type: application/json
I1020 11:03:46.262841 1675 request.go:942] Response Body: {"kind":"APIVersions","versions":["v1"],"serverAddressByClientCIDRs":[{"clientCIDR":"0.0.0.0/0","serverAddress":"ip-10-xxx-xxx-xxx.ec2.internal:443"}]}
I'd suspect a network or firewall problem, but simply doing curl to that endpoint do have some response, though lack of permission:
$ curl -k -v -XGET -H "Accept: application/json, */*" -H "User-Agent: kubectl/v1.14.7 (linux/amd64) kubernetes/1861c59" 'https://XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.eks.amazonaws.com/api?timeout=32s'
Note: Unnecessary use of -X or --request, GET is already inferred.
* Trying x.xxx.xxx.xx:443...
* TCP_NODELAY set
* Connected to XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.eks.amazonaws.com (x.xxx.xxx.xx) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Request CERT (13):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
* subject: CN=kube-apiserver
* start date: Oct 17 10:29:43 2019 GMT
* expire date: Oct 16 10:29:44 2020 GMT
* issuer: CN=kubernetes
* SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x55cb670e87b0)
> GET /api?timeout=32s HTTP/2
> Host: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.eks.amazonaws.com
> Accept: application/json, */*
> User-Agent: kubectl/v1.14.7 (linux/amd64) kubernetes/1861c59
>
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
< HTTP/2 403
< audit-id: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
< content-type: application/json
< x-content-type-options: nosniff
< content-length: 188
< date: Sun, 20 Oct 2019 14:56:42 GMT
<
{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"forbidden: User \"system:anonymous\" cannot get path \"/api\"","reason":"Forbidden","details":{},"code":403}
* Connection #0 to host XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.eks.amazonaws.com left intact
Edit - here's my .kube/config, same as .kube/tv-config (some items masked). It's generated by aws eks update-kubeconfig --name <Cluster>
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: <Same as the one in `Certificate authority
` of EKS>
server: https://<PREFIX>.<REGION>.eks.amazonaws.com
name: arn:aws:eks:<REGION>:<AccountId>:cluster/<Cluster>
contexts:
- context:
cluster: arn:aws:eks:<REGION>:<AccountId>:cluster/<Cluster>
user: arn:aws:eks:<REGION>:<AccountId>:cluster/<Cluster>
name: arn:aws:eks:<REGION>:<AccountId>:cluster/<Cluster>
current-context: arn:aws:eks:<REGION>:<AccountId>:cluster/<Cluster>
kind: Config
preferences: {}
users:
- name: arn:aws:eks:<REGION>:<AccountId>:cluster/<Cluster>
user:
exec:
apiVersion: client.authentication.k8s.io/v1alpha1
args:
- --region
- <REGION>
- eks
- get-token
- --cluster-name
- <Cluster>
command: aws
Wow, it turns out to be the DNS resolution problem, despite that I used the newly install the system for several days without noticing it.
Previously I just tried getent hosts <DNS> for DNS resolution and use curl -v <PREFIX>.<REGION>.eks.amazonaws.com> for DNS resolution test. They both had replied correctly before I found my /etc/resolv.conf is actually empty.
I have missed configuring systemd's DNS resolution. As documented here:
Note that if you want to take advantage of automatic DNS configuration from DHCP, you need to enable systemd-resolved and symlink /run/systemd/resolve/resolv.conf to /etc/resolv.conf
After symbolic linking now it just works as expected!
I want to download from SVN server only part of some large file. SVN server is behind Apache HTTP server, so I create HTTP range request. Then got full file content but not requested range part. Server respond with code 200 not 206, so it not supported range requests. How to enable range requests for SVN server?
Is it possible to get only range part from SVN?
What should I change in SVN server config?
May be in Apache server config?
Thanks in advance.
Session log:
request with: Range: bytes=0-199
response with: Accept-Ranges: bytes Content-Length: 678852
$ curl -v -r 0-199 -u l_chayka:***** https://svn2.*******.net/svn/****/tech_research/dvpls/meta.dvpk > meta.dvpk
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Trying 10.128.40.150...
* TCP_NODELAY set
* Connected to svn2.*******.net (10.128.40.150) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: C:/msys64/mingw64/ssl/certs/ca-bundle.crt
CApath: none
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [93 bytes data]
* TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [2580 bytes data]
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
{ [333 bytes data]
* TLSv1.2 (IN), TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
} [70 bytes data]
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.2 (OUT), TLS handshake, Finished (20):
} [16 bytes data]
* TLSv1.2 (IN), TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
* subject: C=CY; L=*****; O=***** Group Limited; CN=*.******.net
* start date: Jun 2 00:00:00 2017 GMT
* expire date: Jul 14 12:00:00 2020 GMT
* subjectAltName: host "svn2.******.net" matched cert's "*.******.net"
* issuer: C=US; O=DigiCert Inc; OU=www.digicert.com; CN=DigiCert SHA2 High Assurance Server CA
* SSL certificate verify ok.
* Server auth using Basic with user 'l_chayka'
} [5 bytes data]
> GET /svn/****/tech_research/dvpls/meta.dvpk HTTP/1.1
> Host: svn2.******.net
> Authorization: Basic bF9aGF5a2Rgd5E6VzI0MTkwODZnMTk4NDE=
> Range: bytes=0-199
> User-Agent: curl/7.62.0
> Accept: */*
>
{ [5 bytes data]
< HTTP/1.1 200 OK
< Date: Mon, 05 Nov 2018 09:50:00 GMT
< Server: Apache/2.4.7 (Ubuntu)
< Strict-Transport-Security: max-age=63072000; includeSubdomains;
< Last-Modified: Thu, 01 Nov 2018 12:29:28 GMT
< ETag: "617837//tech_research/dvpls/meta.dvpk"
< Cache-Control: max-age=604800
< Accept-Ranges: bytes
< Content-Length: 678852
< X-Content-Type-Options: nosniff
< X-Frame-Options: sameorigin
< Content-Type: application/octet-stream
<
{ [5 bytes data]
100 662k 100 662k 0 0 10.2M 0 --:--:-- --:--:-- --:--:-- 10.2M
* Connection #0 to host svn2.******.net left intact
I'm trying to follow the xively cURL tutorial. I have created new device in develop mode, copied the default auto-generated API key (with READ, WRITE, CREATE, DELETE permissions) to the example in tutorial and get response:
{"title":"Forbidden","errors":"You do not have the necessary permissions to access this resource"}
I must be missing some obvious step. Do I need to activate the API key somehow before using it in scripts?
The cURL command:
curl --request POST \
--data '{"title":"My feed", "version":"1.0.0"}' \
--header "X-ApiKey: cPHLfGw1WJdMAbU8FzbfsdFyJ8suayHEH3OChRrkpYwQCmrb" \
--verbose \
https://api.xively.com/v2/feeds
full verbose output:
* About to connect() to api.xively.com port 443 (#0)
* Trying 216.52.233.120...
* Connected to api.xively.com (216.52.233.120) port 443 (#0)
* successfully set certificate verify locations:
* CAfile: none
CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Server key exchange (12):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using ECDHE-RSA-RC4-SHA
* Server certificate:
* subject: C=US; postalCode=01801; ST=MA; L=Woburn; street=First Floor; street=500 Unicorn Park Drive; O=LogMeIn Inc.; OU=Secure Link SSL Wildcard; CN=*.xively.com
* start date: 2013-05-07 00:00:00 GMT
* expire date: 2014-04-27 23:59:59 GMT
* subjectAltName: api.xively.com matched
* issuer: C=US; O=Network Solutions L.L.C.; CN=Network Solutions Certificate Authority
* SSL certificate verify ok.
> POST /v2/feeds HTTP/1.1
> User-Agent: curl/7.29.0
> Host: api.xively.com
> Accept: */*
> X-ApiKey: cPHLfGw1WJdMAbU8FzbfsdFyJ8suayHEH3OChRrkpYwQCmrb
> Content-Length: 38
> Content-Type: application/x-www-form-urlencoded
>
* upload completely sent off: 38 out of 38 bytes
< HTTP/1.1 403 Forbidden
< Date: Sat, 30 Nov 2013 11:03:15 GMT
< Content-Type: application/json; charset=utf-8
< Content-Length: 98
< Connection: keep-alive
< X-Request-Id: 6cbb9676b448a4967187271dd246b423f7da2e39
<
* Connection #0 to host api.xively.com left intact
{"title":"Forbidden","errors":"You do not have the necessary permissions to access this resource"}
It depends what you are trying to do. POST requests to api.xively.com/v2/feeds are no longer supported with any API key. This is because, since the implementation of devices, programatic creation of feeds directly is no longer supported.
This is an oversight in the Xively tutorial and I will inform the appropriate people to make sure that it gets changed.
In the mean time, since you have already created a device, you are basically ready to start the cURL tutorial at step 3 "Update a Feed". Use the API key and Feed ID from the development device you have already created on the website. Make sure to change your URL, body, and from a POST request to a PUT request.
I think you need to add your feed id onto the end of the URL.
I've executed the following command with the respective response.
However, if I try to get information about the /user, it works, witch means that my token is valid.
What Am I doing wrong?
guto#willie:~/$ curl -v -XPOST -H 'Authorization: token S3CR3T' -H 'Content-Type: application/json; charset=utf-8' https://api.github.com/user/repos -d '{"name":"my-new-repo","description":"my new repo description"}'
Output:
* About to connect() to api.github.com port 443 (#0)
* Trying 207.97.227.243... connected
* Connected to api.github.com (207.97.227.243) port 443 (#0)
* successfully set certificate verify locations:
* CAfile: none
CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using AES256-SHA
* Server certificate:
* subject: O=*.github.com; OU=Domain Control Validated; CN=*.github.com
* start date: 2009-12-11 05:02:36 GMT
* expire date: 2014-12-11 05:02:36 GMT
* subjectAltName: api.github.com matched
* issuer: C=US; ST=Arizona; L=Scottsdale; O=GoDaddy.com, Inc.; OU=http://certificates.godaddy.com/repository; CN=Go Daddy Secure Certification Authority; serialNumber=07969287
* SSL certificate verify ok.
> POST /user/repos HTTP/1.1
> User-Agent: curl/7.21.3 (x86_64-pc-linux-gnu) libcurl/7.21.3 OpenSSL/0.9.8o zlib/1.2.3.4 libidn/1.18
> Host: api.github.com
> Accept: */*
> Authorization: token S3CR3T
> Content-Type: application/json; charset=utf-8
> Content-Length: 62
>
< HTTP/1.1 404 Not Found
< Server: nginx/1.0.4
< Date: Tue, 27 Dec 2011 03:45:12 GMT
< Content-Type: application/json; charset=utf-8
< Connection: keep-alive
< Status: 404 Not Found
< X-RateLimit-Limit: 5000
< ETag: "31b00b4920d3470b70611b10e0ba62a7"
< X-OAuth-Scopes: public_repo, user
< X-RateLimit-Remaining: 4976
< X-Accepted-OAuth-Scopes: repo
< Content-Length: 29
<
{
"message": "Not Found"
}
* Connection #0 to host api.github.com left intact
* Closing connection #0
* SSLv3, TLS alert, Client hello (1):
guto#willie:~/projetos/apostilas/4linux-helper$
Check the Oath Scope GitHub documentation:
$ curl -H "Authorization: bearer TOKEN" https://api.github.com/users/technoweenie -I
HTTP/1.1 200 OK
X-OAuth-Scopes: repo, user
X-Accepted-OAuth-Scopes: user
You need to have the repo scope in order to have the right to create a repo, as illustrated by the SO question "Github v3 API - create a REPO".
repo
DB read/write access, and Git read access to public and private repos.
NOTE: Your application can request the scopes in the initial redirection.
You can specify multiple scopes by separating them by a comma.
https://github.com/login/oauth/authorize?
client_id=...&
scope=user,public_repo
try using
curl -v -X POST -H 'Content-Type: application/x-www-form-urlencoded' -d '{"name":"my-new-repo","description":"my new repo description"}' https://api.github.com/user/repos?access_token=S3CR3T