I am using a CURL command from WINDOWS.
I have posted the command below.
I have also posted the results of the command.
It appears to me, the server is terminating the connection before sending the certificate.
But I could wrong.
Any ideas why the certificate is not being sent?
curl -v --cert C:\Users\myFolder\Downloads\hs_test_cert.pem --key C:\Users\myFolder\Downloads\hst_test_key.pem https://myAPIGEEhost.apigee.net/MyBackendName_CCDA_API?apikey=MyAssignedAPIKey
* Trying 00.00.00...
* TCP_NODELAY set
* Connected to myAPIGEEhost.apigee.net (00.00.00) port 443 (#0)
* schannel: SSL/TLS connection with myAPIGEEhost.apigee.net port 443 (step 1/3)
* schannel: checking server certificate revocation
* schannel: sending initial handshake data: sending 203 bytes...
* schannel: sent initial handshake data: sent 203 bytes
* schannel: SSL/TLS connection with myAPIGEEhost.apigee.net port 443 (step 2/3)
* schannel: failed to receive handshake, need more data
* schannel: SSL/TLS connection with myAPIGEEhost.apigee.net port 443 (step 2/3)
* schannel: encrypted data got 4096
* schannel: encrypted data buffer: offset 4096 length 4096
* schannel: encrypted data length: 4026
* schannel: encrypted data buffer: offset 4026 length 4096
* schannel: received incomplete message, need more data
* schannel: SSL/TLS connection with myAPIGEEhost.apigee.net port 443 (step 2/3)
* schannel: encrypted data got 964
* schannel: encrypted data buffer: offset 4990 length 5050
* schannel: a client certificate has been requested
* schannel: SSL/TLS connection with myAPIGEEhost.apigee.net port 443 (step 2/3)
* schannel: encrypted data buffer: offset 4990 length 6014
* schannel: sending next handshake data: sending 133 bytes...
* schannel: SSL/TLS connection with myAPIGEEhost.apigee.net port 443 (step 2/3)
* schannel: encrypted data got 274
* schannel: encrypted data buffer: offset 274 length 6014
* schannel: SSL/TLS handshake complete
* schannel: SSL/TLS connection with myAPIGEEhost.apigee.net port 443 (step 3/3)
* schannel: stored credential handle in session cache
> GET /MyBackendName_CCDA_API?apikey=MyAssignedAPIKey HTTP/1.1
> Host: myAPIGEEhost.apigee.net
> User-Agent: curl/7.55.1
> Accept: */*
>
* schannel: client wants to read 102400 bytes
* schannel: encdata_buffer resized 103424
* schannel: encrypted data buffer: offset 0 length 103424
* schannel: encrypted data got 421
* schannel: encrypted data buffer: offset 421 length 103424
* schannel: decrypted data length: 361
* schannel: decrypted data added: 361
* schannel: decrypted data cached: offset 361 length 102400
* schannel: encrypted data length: 31
* schannel: encrypted data cached: offset 31 length 103424
* schannel: server closed the connection
* schannel: schannel_recv cleanup
* schannel: decrypted data returned 361
* schannel: decrypted data buffer: offset 0 length 102400
< HTTP/1.1 400 Bad Request
< Date: Mon, 27 Sep 2021 18:22:21 GMT
< Content-Type: text/html
< Content-Length: 231
< Connection: close
<
<html>
<head><title>400 No required SSL certificate was sent</title></head>
<body>
<center><h1>400 Bad Request</h1></center>
<center>No required SSL certificate was sent</center>
<hr><center>server</center>
</body>
</html>
* Closing connection 0
* schannel: shutting down SSL/TLS connection with myAPIGEEhost.apigee.net port 443
* schannel: clear security context handle
It appears to me, the server is terminating the connection before sending the certificate. But I could wrong.
The TLS handshake of the HTTPS connection succeeded, since otherwise the server could not have sent a HTTP response - which it clearly did.
* schannel: a client certificate has been requested
This debug message indicates that the server requested a certificate from the client, i.e. mutual authentication. This certificate requests comes after the server has already send its own certificate to the client.
<center>No required SSL certificate was sent</center>
While you have provided a client certificate and key on the command line none was actually send. The reason is that client certificates are not supported with SChannel in curl 7.55.1 (what you are using), see TODO:
15.1 Add support for client certificate authentication
WinSSL/SChannel currently makes use of the OS-level system and user
certificate and private key stores. This does not allow the application
or the user to supply a custom client certificate using curl or libcurl.
Therefore support for the existing -E/--cert and --key options should be
implemented by supplying a custom certificate to the SChannel APIs, see:
- Getting a Certificate for Schannel
https://msdn.microsoft.com/en-us/library/windows/desktop/aa375447.aspx
Better support for client certificates were added in newer version but even with the current latest (7.79. ) it is not possible to give a PEM file as cert or key with the SChannel TLS backend.
Related
I read the lot of blogs about the issue but none of the workaround /solutions worked for me.
I am using the curl command like below
curl -v https://golang.org/dl/?mode=json
* Trying 142.250.80.113:443...
* Connected to golang.org (142.250.80.113) port 443 (#0)
* schannel: disabled automatic use of client certificate
* ALPN: offers http/1.1
* schannel: next InitializeSecurityContext failed: Unknown error (0x80092012) - The revocation function was unable to check revocation for the certificate.
* Closing connection 0
curl: (35) schannel: next InitializeSecurityContext failed: Unknown error (0x80092012) - The revocation function was unable to check revocation for the certificate.
I changed the setting in gitbash (windows) to use openssl using the below command
git config --global http.sslBackend "openssl"
I am getting the below error after changing openssl
$ curl -v https://golang.org/dl/?mode=json
* Trying 172.253.62.141:443...
* Connected to golang.org (172.253.62.141) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* CAfile: C:/Program Files/Git/mingw64/ssl/certs/ca-bundle.crt
* CApath: C:\Users\AL25229
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS alert, unknown CA (560):
* SSL certificate problem: self signed certificate in certificate chain
* Closing connection 0
curl: (60) SSL certificate problem: self signed certificate in certificate chain
More details here: https://curl.se/docs/sslcerts.html
curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
I have the CA cert/pem file which my organization provides. I am getting all those error messages when I connect with VPN. Connecting VPN is mandatory. By disabling the VPN, it works fine.
Recently I've been evaluating different API Gateway (API GW) options for the IoT-based project. The purpose of this was to find a good enough solution for performing Mutual TLS (mTLS) authentication of the devices and API GW.
Most of the solutions I've tried out seem to perform mTLS during the TLS handshake as nicely depicted here. So this is what I understand OSI Layer 4 (TCP/IP) authentication method.
However, the Kong API Gateway seem to do it at OSI Layer 7 (Application). Basically, no client auth during the TLS handshake phase, and rather application layer validates the peer certificate. Hence it's able to send the response with 401 status and some payload (which is not possible, if TLS handshake fails). Example
√ poc-mtls-local-env % make test-fail-wrong-cert master
curl -v --cacert certs/gen/ca-chain.crt \
--key certs/gen/device.key \
--cert certs/gen/device.crt \
https://mtls.auth.local.com/echo
* Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to mtls.auth.local.com (127.0.0.1) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: certs/gen/ca.crt
CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Request CERT (13):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS handshake, CERT verify (15):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use h2
* Server certificate:
* subject: C=US; ST=NY; L=NYC; O=Sample; OU=UDS; CN=local.com
* start date: Jul 29 12:10:25 2021 GMT
* expire date: Jul 29 12:10:25 2022 GMT
* subjectAltName: host "mtls.auth.local.com" matched cert's "mtls.auth.local.com"
* issuer: C=US; ST=NY; O=Sample; OU=UDS; CN=Sample Intermediate CA; emailAddress=it#sample.com
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7fc0dd808200)
> GET /echo HTTP/2
> Host: mtls.auth.local.com
> User-Agent: curl/7.64.1
> Accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
< HTTP/2 401
< date: Tue, 10 Aug 2021 06:46:13 GMT
< content-type: application/json; charset=utf-8
< content-length: 49
< x-kong-response-latency: 4
< server: kong/2.4.1.1-enterprise-edition
<
* Connection #0 to host mtls.auth.local.com left intact
{"message":"TLS certificate failed verification"}* Closing connection 0
We can clearly see that request goes past the TLS handshake successfully, and the application layer forms 401 response with {"message": "TLS certificate failed verification"}.
This made me think of the following questions:
Formally speaking, can it also be called mTLS what Kong does here?
Are there any potential pitfalls with such an approach?
Most of the solutions I've tried out seem to perform mTLS during the TLS handshake as nicely depicted here. So this is what I understand OSI Layer 4 (TCP/IP) authentication method.
Since TLS is above layer OSI layer 4 the authentication is also above layer 4. But OSI layers aside (which don't sufficiently match today's reality above layer 4 anyway) you essentially ask at what stage the mutual authentication happens.
Mutual authentication in TLS happens in two stages: requesting the clients certificate and validating that the certificate matches the requirements. Requesting the certificate is always done inside the TLS handshake, although it does not need to be the initial TLS handshake of the connection.
Validating the certificate can be done inside the TLS handshake, outside of it or a combination of both. Typically it is checked inside the handshake that the certificate is issued by some trusted certificate authority, but further checks for a specific subject or so might be application specific and will thus be done after the TLS handshake inside the application. But it might also be that the full validation is done inside or outside the TLS handshake.
Accepting any certificates inside the TLS handshake and validating the certificate then outside the handshake only, has the advantage that one can return a useful error message to the client inside the established TLS connection. Validation errors inside the TLS handshake instead result in cryptic errors like handshake error alerts or just closing the connection, which are not that helpful to debug the problem.
Artifactory version:
EnterpriseX license 7.15.3 rev 71503900
I have admin level access on the target repository. I'm trying to upload an artifact which is close to 10GB long (air-gapped .tgz) file. I'm getting the following error.
When I try to upload the file using Artifactory UI (clicking Deploy button on the target repo), It kicks me out back to login screen, while the upload says 100% complete but check-box/options for deploying artifact as "bundle artifact" or as per user defined "Layout" is not showing up, Deploy button at the bottom of that pop-up window is also grayed out.
Tried curl using Access Key as per documentation here: https://www.jfrog.com/confluence/display/JFROG/Artifactory+REST+API but it gives me an error mesg after a very long time (12-13+ hours) about 502 Bad Gateway.
I also tried using -x "" option to bypass proxy or prefixed the same command with no_proxy="10.20.30.40" curl .... but getting the same error. Tried -u user:pass, gives same error. Artifactory settings/DB configuration has been changed to allow for more connections and longer time allotted before timeout but still getting the same error.
PS: Notice that 0 under 258M in the 2nd line, i.e. line under We are completely uploaded and fine that tells me, nothing is getting uploaded.
$ curl -H "X-JFrog-Art-Api:akfljkljkALDJALKDALKJDALKDJLASJDLAKDJALKDJALKDJALKDJALKDJALJKDLKAJFLANCMNLLgoEjcfZ-c7v58FmyaAUsJ8c0gFV6VVHp2WpvYbU7IftRyzirHEmsGJ3MRL0eZqCkyZYI_pkrcgXb3H2QcQ6RxDpbY2UYgX5AKQlrLhtb644wlBtK1VelsJ90d-6TPrr59ss-igGDhS-HUpSMAYMBl9cXQtT5hAR8Q" -X PUT "https://artifactory.company.com/artifactory/AlphaPipeline-PRJ-ProjectABCPipeline-Production-Local/Pipeline-release-3.0.0.tgz" -T Pipeline-release-3.0.0.tgz -v
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Trying 10.20.30.40...
* TCP_NODELAY set
* Connected to artifactory.company.com (10.20.30.40) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:#STRENGTH
* successfully set certificate verify locations:
* CAfile: C:/tools/installed/Git/mingw64/ssl/certs/ca-bundle.crt
CApath: none
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
} [5 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.2 (IN), TLS handshake, Server hello (2):
{ [76 bytes data]
* TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [4146 bytes data]
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
{ [333 bytes data]
* TLSv1.2 (IN), TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
} [70 bytes data]
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
} [1 bytes data]
* TLSv1.2 (OUT), TLS handshake, Finished (20):
} [16 bytes data]
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
{ [1 bytes data]
* TLSv1.2 (IN), TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.2 / ABCDE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
* subject: C=US; ST=Denver; L=Centennial; O=Spotair Technologies; CN=artifactory.company.com
* start date: Feb 24 20:54:47 2021 GMT
* expire date: Feb 24 20:54:46 2022 GMT
* subjectAltName: host "artifactory.company.com" matched cert's "artifactory.company.com"
* issuer: C=US; O=Entrust, Inc.; OU=See www.entrust.net/legal-terms; OU=(c) 2012 Entrust, Inc. - for authorized use only; CN=Entrust Certification Authority - L1K
* SSL certificate verify ok.
} [5 bytes data]
> PUT /artifactory/AlphaPipeline-PRJ-ProjectABCPipeline-Production-Local/Pipeline-release-3.0.0.tgz HTTP/1.1
> Host: artifactory.company.com
> User-Agent: curl/7.60.0
> Accept: */*
> X-JFrog-Art-Api:akfljkljkALDJALKDALKJDALKDJLASJDLAKDJALKDJALKDJALKDJALKDJALJKDLKAJFLANCMNLLgoEjcfZ-c7v58FmyaAUsJ8c0gFV6VVHp2WpvYbU7IftRyzirHEmsGJ3MRL0eZqCkyZYI_pkrcgXb3H2QcQ6RxDpbY2UYgX5AKQlrLhtb644wlBtK1VelsJ90d-6TPrr59ss-igGDhS-HUpSMAYMBl9cXQtT5hAR8Q
> Content-Length: 11629208547
> Expect: 100-continue
>
{ [5 bytes data]
< HTTP/1.1 100 Continue
} [5 bytes data]
99 10.8G 0 0 99 10.7G 0 357M 0:00:31 0:00:30 0:00:01 258M* We are completely uploaded and fine
100 10.8G 0 0 100 10.8G 0 239k 13:11:33 13:11:33 --:--:-- 0{ [5 bytes data]
< HTTP/1.1 502 Bad Gateway
< Date: Wed, 07 Jul 2021 18:55:33 GMT
< Server: Apache/2.4.46 (Unix) OpenSSL/1.1.1k
< Content-Length: 232
< Content-Type: text/html; charset=iso-8859-1
<
{ [232 bytes data]
100 10.8G 0 232 100 10.8G 0 239k 13:11:33 13:11:33 --:--:-- 54<!DOCTYPE HTML PUBLIC "-//IETF//DPRJ HTML 2.0//EN">
<html><head>
<title>502 Bad Gateway</title>
</head><body>
<h1>Bad Gateway</h1>
<p>The proxy server received an invalid
response from an upstream server.<br />
</p>
</body></html>
* Connection #0 to host artifactory.company.com left intact
Another odd/annoying thing: If I use the same API Access Key on a small file (like few KBs) then I instantly hit the following 403 Bad props auth token error. If I pass another file (ex: > 2MB) without changing anything in the command, it gives me the above 502 Bad Gateway error.
{ [824 bytes data]
100 2576 100 824 100 1752 675 1437 0:00:01 0:00:01 --:--:-- 2113{
"errors" : [ {
"status" : 403,
"message" : "Bad props auth token: apiKey=my_long_apiAccessKeyHere."
} ]
}
PS: Related post when using Artifactory GUI to upload the file.
Artifactory - Can't upload or see DEPLOY button getting highlighted - can't upload artifact .tar .tgz
One thing to notice is the API key being used, I see this in the header,
"X-JFrog-Art-Api:akfljkljkALDJALKDALKJDALKDJLASJDLAKDJALKDJALKDJALKDJALKDJALJKDLKAJFLANCMNLLgoEjcfZ-c7v58FmyaAUsJ8c0gFV6VVHp2WpvYbU7IftRyzirHEmsGJ3MRL0eZqCkyZYI_pkrcgXb3H2QcQ6RxDpbY2UYgX5AKQlrLhtb644wlBtK1VelsJ90d-6TPrr59ss-igGDhS-HUpSMAYMBl9cXQtT5hAR8Q"
From where did you generated the API key? As a test, can you use the credentials as -u :.
Also, generate the API key from the profile page and use that in the header.
I recently installed k3s on debian 10 and I am having problems connecting to the internet from inside the pods. Ping, DNS, HTTP, works but TLS is having problems with the Handshake. I made a tcpdump and see ClientHello's and then immediatly a response from the server with internal error SSL error 80. On the VM where cluster runs on everything works fine, just not inside the pods.
Example in container:
pacman -Syu
:: Synchronizing package databases...
error: failed retrieving file 'core.db' from mirror.pkgbuild.com : error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error
error: failed retrieving file 'core.db' from mirror.rackspace.com : error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error
error: failed retrieving file 'core.db' from mirror.leaseweb.net : error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error
error: failed to update core (download library error)
error: failed retrieving file 'extra.db' from mirror.pkgbuild.com : error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error
error: failed retrieving file 'extra.db' from mirror.rackspace.com : error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error
error: failed retrieving file 'extra.db' from mirror.leaseweb.net : error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error
error: failed to update extra (download library error)
error: failed retrieving file 'community.db' from mirror.pkgbuild.com : error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error
error: failed retrieving file 'community.db' from mirror.rackspace.com : error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error
error: failed retrieving file 'community.db' from mirror.leaseweb.net : error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error
error: failed to update community (download library error)
error: failed to synchronize all databases
curl -4 -v https://www.google.com
* Trying 157.230.127.168:443...
* Connected to www.google.com (157.230.127.168) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
* CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS alert, internal error (592):
* error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error
* Closing connection 0
curl: (35) error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error
On VM:
curl -4 -v https://www.google.com
* Expire in 0 ms for 6 (transfer 0x556e74843fb0)
<snip>A lot of these expire messages</snip>
* Expire in 10 ms for 1 (transfer 0x556e74843fb0)
* Trying 172.217.23.100...
* TCP_NODELAY set
* Expire in 200 ms for 4 (transfer 0x556e74843fb0)
* Connected to www.google.com (172.217.23.100) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: none
CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
* subject: C=US; ST=California; L=Mountain View; O=Google LLC; CN=www.google.com
* start date: Mar 11 15:00:19 2021 GMT
* expire date: Jun 3 15:00:18 2021 GMT
* subjectAltName: host "www.google.com" matched cert's "www.google.com"
* issuer: C=US; O=Google Trust Services; CN=GTS CA 1O1
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x556e74843fb0)
> GET / HTTP/2
> Host: www.google.com
> User-Agent: curl/7.64.0
> Accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
< HTTP/2 200
< date: Fri, 26 Mar 2021 01:32:52 GMT
< expires: -1
< cache-control: private, max-age=0
< content-type: text/html; charset=ISO-8859-1
< p3p: CP="This is not a P3P policy! See g.co/p3phelp for more info."
< server: gws
< x-xss-protection: 0
< x-frame-options: SAMEORIGIN
< set-cookie: NID=212=0Nbn1MMUvMRUNa8iBWgekw-YrWof3yeeBb22v94ZrQ4KkMeVm5wouwkK9ElA353VmZCp_TOXah6KfC_KQX7S48W-IgXlQUz1z4ytqKnDzSXM7X40rLw8tBwMi4oH7eLyE4nGGAfsSlUne28lwMNLobMxYls54iUbgM4x9-kuMCA; expires=Sat, 25-Sep-2021 01:32:52 GMT; path=/; domain=.google.com; HttpOnly
< alt-svc: h3-29=":443"; ma=2592000,h3-T051=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"
< accept-ranges: none
< vary: Accept-Encoding
<hmtl>...,</html>
Does anybody have an idea how I can debug this or fix it?
UPDATE
Here a curl from inside the VM and a pod. I fixed the Webserver to a known working one on the Internet (Vercel Hosted Website) aswell so we debug the same server. But it is always the same error inside the pod. no matter which server I contact.
kubectl run -i --tty --rm debug --image=archlinux --restart=Never -- bash
Pod:
[root#debug /]# curl -v --tlsv1.2 --tls-max 1.2 https://wegmueller.it/ > /dev/null
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Trying 157.230.127.168:443...
* Connected to wegmueller.it (157.230.127.168) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
* CApath: none
} [5 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
} [229 bytes data]
* TLSv1.2 (IN), TLS alert, internal error (592):
{ [2 bytes data]
* error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
* Closing connection 0
curl: (35) error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error
[root#debug /]#
VM:
curl -v --tlsv1.2 --tls-max 1.2 https://wegmueller.it/ > /dev/null
* Expire in 0 ms for 6 (transfer 0x559d896cafb0)
* Expire in 1 ms for 1 (transfer 0x559d896cafb0)
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Expire in 0 ms for 1 (transfer 0x559d896cafb0)
* Expire in 1 ms for 1 (transfer 0x559d896cafb0)
<snip a ton of expire messages>
* Expire in 50 ms for 1 (transfer 0x559d896cafb0)
* Trying 76.76.21.21...
* TCP_NODELAY set
* Expire in 200 ms for 4 (transfer 0x559d896cafb0)
* Connected to wegmueller.it (76.76.21.21) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: none
CApath: /etc/ssl/certs
} [5 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
} [223 bytes data]
* TLSv1.2 (IN), TLS handshake, Server hello (2):
{ [106 bytes data]
* TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [2458 bytes data]
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
{ [300 bytes data]
* TLSv1.2 (IN), TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
} [37 bytes data]
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.2 (OUT), TLS handshake, Finished (20):
} [16 bytes data]
* TLSv1.2 (IN), TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use h2
* Server certificate:
* subject: CN=wegmueller.it
* start date: Mar 16 21:08:10 2021 GMT
* expire date: Jun 14 21:08:10 2021 GMT
* subjectAltName: host "wegmueller.it" matched cert's "wegmueller.it"
* issuer: C=US; O=Let's Encrypt; CN=R3
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
} [5 bytes data]
* Using Stream ID: 1 (easy handle 0x559d896cafb0)
} [5 bytes data]
> GET / HTTP/2
> Host: wegmueller.it
> User-Agent: curl/7.64.0
> Accept: */*
>
{ [5 bytes data]
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
} [5 bytes data]
< HTTP/2 200
< date: Sun, 28 Mar 2021 15:24:59 GMT
< content-type: text/html; charset=utf-8
< content-disposition: inline; filename="index"
< cache-control: public, max-age=0, must-revalidate
< content-length: 3154
< access-control-allow-origin: *
< etag: W/"637eb49039bc1f6a7aa9e4d88a56b0708b722983f403c3c2f717371b2b25a472"
< accept-ranges: bytes
< x-vercel-cache: MISS
< age: 0
< server: Vercel
< x-vercel-id: cdg1::z6h9l-1616945098788-71384c5744be
< strict-transport-security: max-age=63072000
<
{ [1112 bytes data]
100 3154 100 3154 0 0 3604 0 --:--:-- --:--:-- --:--:-- 3600
* Connection #0 to host wegmueller.it left intact
error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error
openssl error code contains 3 parts.
First 8 bits is module serial number. 14(HEX) means ssl module which has serial number 20.
The next 12 bits is function SN. 094(HEX) is SSL_SRP_CTX_init func with SN 148(DEC).
The next 12 bits 438(HEX) is error reason SSL_R_TLSV1_ALERT_INTERNAL_ERROR with 1080(DEC).
To sum up,code 14094438 means that server error occurred when it tried to initiate SSL context.
But what's the reason?
client side
client side send CLIENT HELLO with its tls version, ciphers suite and etc to server side.
what's the difference of 2 CLIENT HELLO packets on pods or VM? This need to be clear.
Here you use newest version tls 1.3. For more infomation, you may downgrade client version.
curl has options to control the TLS version used. if you want to specify that TLS 1.2 is used but not 1.1 or 1.3 etc, you need something like
curl --tlsv1.2 --tls-max 1.2 ...
server side
I also noticed that www.google.com (157.230.127.168) is different from www.google.com (172.217.23.100).
Does server side's difference lead to this situation? We can confirm with sending a curl to the same IP.
I assume either you and/or the server uses an unmatched openssl library.But before drawing conclusion, we need more debug info.
I am trying to send a simple curl request:
curl -k -i --key ./key.pem --cert ./cert.pem https://target_ip/whatever/
The problem I'm having is that it does not send any certificate. The validation clearly passes as otherwise I was getting errors such as the key does not match but then I can see in wireshark that the certificates are not being sent in the TCP connection around Client Hello. Switches like --verbose or --cacert don't do much either.
I am able to send the very same certificates through postman successfully.
I have tried sending the same curl request from various sources such as my WSL2 ubuntu, a debian container in the cloud, a VM, ...
Any tips why it is not sending the certs?
EDIT I - output from curl -v
* Trying 52.xxx.xxx.xx:443...
* TCP_NODELAY set
* Connected to 52.xxx.xxx.xx (52.xxx.xxx.xx) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
* subject: CN=NGINXIngressController
* start date: Aug 10 18:08:13 2020 GMT
* expire date: Aug 10 18:08:13 2021 GMT
* issuer: CN=NGINXIngressController
* SSL certificate verify result: self signed certificate (18), continuing anyway.
> GET /whatever/ HTTP/1.1
> Host: custom.localhost.dev
> User-Agent: curl/7.68.0
> Accept: */*
> Authorization: Bearer eyJ0...
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 403 Forbidden
HTTP/1.1 403 Forbidden
< Server: nginx/1.19.0
Server: nginx/1.19.0
< Date: Mon, 10 Aug 2020 22:23:24 GMT
Date: Mon, 10 Aug 2020 22:23:24 GMT
< Content-Type: text/html
Content-Type: text/html
< Content-Length: 153
Content-Length: 153
< Connection: keep-alive
Connection: keep-alive
<
<html>
<head><title>403 Forbidden</title></head>
<body>
<center><h1>403 Forbidden</h1></center>
<hr><center>nginx/1.19.0</center>
</body>
</html>
* Connection #0 to host 52.xxx.xxx.xx left intact
EDIT II - wireshark captures
It seems to be too much of a hassle to anonymise pcap, so here's just some snaps. Hopefully you'll be able to see all you need. I have highlighted the packet where I do (not) see the cert being sent. Note that I am running the postman on my windows workstation, whereas the curl is in the WSL2, hence the different source addresses. Other hosts for curl did behave the same though.
Curl
Postman
EDIT III - Client Hellos
Curl
Postman
The ClientHello shows a clear difference: postman uses the server_name extension (SNI) to provide the expected hostname while curl does not.
This likely triggers a different part of the configuration in the web server: postman triggers access to the specific virtual host given as server_name while curl will probably run into the default configuration. Assuming that only the specific virtual host enables client certificates this explains why the CertificateRequest is send by the server only to postman but not to curl.
It is unclear what this hostname is, but based on the length it cannot be an IP address. Thus postman somehow must know the expected hostname of the server even though it is claimed that the access was done with https://target_ip/ only, i.e. without a given hostname. curl cannot derive from this URL the expected hostname and thus cannot set server_name. To make curl be aware of the hostname to set server_name while still being able to access a specific IP use the --resolve option:
curl --resolve hostname:443:target_ip https://hostname/