Get net/http: TLS handshake timeout golang - ssl

Need your help with this
package main
import (
"log"
"net/http"
)
func main() {
client := &http.Client{}
_, err := client.Get("https://www.marathonbet.com/en/")
if err != nil {
log.Fatalf("%s\n", err)
}
}
This always return:Get https://www.marathonbet.com/en/: net/http: TLS handshake timeout exit status 1
I`m try:
it
and use this lib
and do it
But nothing works for me..
So, please help me.
Update:
In Python 2.7 with requests this works:
s = Session()
r = s.get('https://www.marathonbet.com/en/, verify=False)
But i need do it with go(
UPD:
Fixed: Just replase https://www.marathonbet.com/en/ to https://87.117.250.213/en/ and adding skip verify.
thx all for help.

Here's a more detailed explanation of what's happening. When you call a domain, your HTTP client calls a DNS server. The DNS server responds with the IP of the target server. At this point everything's OK.
If it's an HTTPS connection, then it starts the TLS handshake. Here's how it works.
And this the point where you experienced the issue. The request was sent but the server didn't answer correctly (or at all). It may be caused by many factors like the server:
isn't accessible
needs more time to respond
can be hidden behind some firewall/proxy that refuses the connection
block all requests from your IP/location
etc
By providing the skip verify option and providing the explicit IP address, you skips everything I described above. It means:
if the server's IP changes your code will stop working
if someone perform a man-in-the-middle attach you won't find out about it.
It's hard to find out what's the root cause without a more deep investigation. If you want to find out what's happening, use the httptrace as #Flowchartsman suggested

Related

Python's default SSL certificate context not working in requests method when behind proxy, works fine otherwise

I have the below function in my code, which works perfectly fine when I'm not behind any proxy. In fact, without even mentioning the certifi default CA certificate, it works fine if I pass verify=TRUE, I guess, because it works in the same way.
def reverse_lookup(lat, long):
cafile=certifi.where()
params={'lat' : float(lat), 'lon' : float(long), 'format' : 'json',
'accept-language' : 'en', 'addressdetails' : 1}
response = requests.get("https://nominatim.openstreetmap.org/reverse", params=params, verify=cafile)
#response = requests.get("https://nominatim.openstreetmap.org/reverse", params=params, verify=True) <-- this works as well
result = json.loads(response.text)
return result['address']['country'], result['address']['state'], result['address']['city']
When I run the same code from within my enterprise infrastructure (where I'm behind proxy), I make some minor changes in the code mentioning the proxy as parameter in requests method:
def reverse_lookup(lat, long):
cafile=certifi.where()
proxies = {"https" : "https://myproxy.com"}
params={'lat' : float(lat), 'lon' : float(long), 'format' : 'json',
'accept-language' : 'en', 'addressdetails' : 1}
response = requests.get("https://nominatim.openstreetmap.org/reverse", params=params, verify=cafile, proxies=proxies)
result = json.loads(response.text)
return result['address']['country'], result['address']['state'], result['address']['city']
But it gives me one out of these 3 SSL errors at different times, if I set verify=True or verify=certifi.where():
CERTIFICATE_VERIFY_FAILED
UNKNOWN_PROTOCOL
WRONG_VERSION_NUMBER
Only time it works is when I completely bypass the SSL verification with verify=False
My questions are:
Since I'm sending the https request via proxy, is it ok if I bypass SSL verification ?
How to make the default context of SSL verification work in this case, when I'm behind proxy ?
Any help is appreciated. Code tested in both Python 2.7.15 and 3.9
Since I'm sending the https request via proxy, is it ok if I bypass SSL verification ?
Do you need the protection offered by HTTPS, i.e. encryption of the application data (like passwords, but also the full URL) to protect against sniffing or modifications by a malicious man in the middle? If you don't need the protection, then you can bypass certificate validation.
How to make the default context of SSL verification work in this case, when I'm behind proxy ?
The proxy is doing SSL interception and when doing this issues a new certificate for this site based on an internal CA. If this is expected (i.e. not an attack) then you need to import the CA from the proxy as trusted with verify='proxy-ca.pem'. Your IT department should be able to provide you with the proxy CA.
But it gives me one out of these 3 SSL errors at different times, if I
set verify=True or verify=certifi.where():
CERTIFICATE_VERIFY_FAILED
UNKNOWN_PROTOCOL
WRONG_VERSION_NUMBER
It should only give you CERTIFICATE_VERIFY_FAILED. The two other errors indicate wrong proxy settings, typically setting https_proxy to https://... instead of http://... (which also can be seen in your code).

SSL redirect changes client IP address read from HTTPResponse

I am using Perfect Framework for my server side application running on an AWS EC2 instance. I am using the following code to get client IP address.
open static func someapi(request: HTTPRequest, _ response: HTTPResponse) {
var clientIP = request.remoteAddress.host }
This was working fine until I installed ssl certificate on my EC2 instance and start redirecting incoming traffic to port 443.
Now this code gives me the ip of my server, i think due to the redirect, Perfect somehow think request comes from itself.
Is there any other method to get client IP address? Or do i have to try something else?
Thanks!
For anyone struggling for the same problem, original client ip can be found in one of the header fields called "xForwardedFor" if there is a redirect, like the following:
var clientIP = request.remoteAddress.host
let forwardInfoResut = request.headers.filter { (item) -> Bool in
item.0 == HTTPRequestHeader.Name.xForwardedFor
}
if let forwardInfo = forwardInfoResut.first {
clientIP = forwardInfo.1
}
Hope this helps somebody, cheers!
Perhaps you should ask the people you are paying for support and whom manage the infrastructure how it works before asking us?
The convention, where an http connection is terminated elsewhere than the server is to inject an x-forwarded-for header. If there is already such a header, the intermediate server injects the client IP address at the front of the list.

Erlang SSL - Certificate not suitable on sni_fun callback

I got this error:
SSL: hello: ssl_handshake.erl:171:Fatal error: internal error - server_has_no_suitable_certificates
when supplying a der-decocded certificate for the callback function in the {sni_fun, CallbackFun} option. The CallbackFun returns [{cacerts, [Cert]}], where Cert is der-encoded. So things comply with Erlang ssl module's documentation.
I tried look into the otp source code. It seems that whatever the callback returns is ignored, thus causing this function clause to be evaluated:
certificate_chain(undefined, _, _) ->
{error, no_cert};
which leads to that error! But I could be wrong as I kind of lost in browsing the code base...
If it helps, I use a self-signed CA to sign CSRs that are generated as a SNI is found during a TLS handshake (via sni_fun option).
please advise! thanks a lot!
Update:
I tried Erlang OTP 20.3 release and get another error:
TLS server: In state hello at tls_connection.erl:739 generated SERVER ALERT: Fatal - Handshake Failure - malformed_handshake_data
Looking at OTP source code, it is result of an exception from this block:
try
Version = ssl_handshake:select_version(tls_record, ClientVersion, Versions),
case ssl_cipher:is_fallback(CipherSuites) of
true ->
Highest = tls_record:highest_protocol_version(Versions),
case tls_record:is_higher(Highest, Version) of
true ->
?ALERT_REC(?FATAL, ?INAPPROPRIATE_FALLBACK);
false ->
handle_client_hello(Version, Hello, SslOpts, Info, Renegotiation)
end;
false ->
handle_client_hello(Version, Hello, SslOpts, Info, Renegotiation)
end
catch
_:_ ->
?ALERT_REC(?FATAL, ?HANDSHAKE_FAILURE, malformed_handshake_data)
end.
I am sure the cert is fine, I could view it as well as convert it between DER/PEM formats with openssl with no error. Is there a way to reveal what kind of exception it is in this case?
I solved the problem: the sni_fun must return the list of
[{cert, DerdecodedCert}, {keyfile, PathToTheCsrKeyFile}]
I was returning only
[{cert, DerdecodedCert}]
(which was so instructed by Erlang ssl doc)
Hope this helps anyone who bumps into similar problem as I did!

Golang: tls.Handshake hangs after connection upgrade

I am writing a TCP text protocol for a project that I am working on. One of the commands in the protocol is STARTTLS which should upgrade the connection to TLS and continue on. My code to upgrade the connection is similar to the answer in this question. The problem I am having is when I am upgrading the TLS connection, the tlsConn.Handshake will hang and never let go. There are some code samples below. Any help is greatly appreciated.
After the STARTTLS command is received...
// Init a new TLS connection. I need a *tls.Conn type
// so that I can do the Handshake()
s.Logf("++> Upgrading connection to TLS")
tlsConn := tls.Server(s.Conn, s.Server.TLSConfig)
s.Logf("++> Attempting TLS Handshake")
tlsConn.Handshake()
s.Logf("++> TLS Handshake Successful")
// Here is the trick. Since I do not need to access
// any of the TLS functions anymore,
// I can convert tlsConn back in to a net.Conn type
s.Conn = net.Conn(tlsConn)
s.Logf("++> Updating read/write buffers")
s.reader = textproto.NewReader(bufio.NewReader(s.Conn))
s.writer = textproto.NewWriter(bufio.NewWriter(s.Conn))
s.Printf("100 SUCCESS")
The client is currently upgrading the connection right after it sends the STARTTLS command like this...
c.conn = tls.Client(c.conn, clientTLSConfig)
The server *tls.Config looks like this...
// Load the key and certificate - paths are provided in flags.
cert, err := tls.LoadX509KeyPair(flagTLSCert, flagTLSKey)
if err != nil {
log.Fatal(err)
}
// Create the TLS config
tlsConfig := &tls.Config{
Certificates: []tls.Certificate{cert},
ClientAuth: tls.VerifyClientCertIfGiven,
ServerName: fqdn(),
}
The client *tls.Config looks like this...
clientTLSConfig := &tls.Config{
InsecureSkipVerify: true,
}
Do you call c.conn.Handshake() or do something else to initiate the TLS handshake on the client side?
If the client does not initiate the handshake by sending TLS Client Hello, the server will sit forever waiting for it.
This is my best guess as you did not provide much of the client side code. Also checking with tcpdump would help to narrow the problem down (to server or client side).

How to do a https request with bad certificate?

Say I want to get https://golang.org programatically. Currently golang.org (ssl) has a bad certificate which is issued to *.appspot.com So when I run this:
package main
import (
"log"
"net/http"
)
func main() {
_, err := http.Get("https://golang.org/")
if err != nil {
log.Fatal(err)
}
}
I get (as I expected)
Get https://golang.org/: certificate is valid for *.appspot.com, *.*.appspot.com, appspot.com, not golang.org
Now, I want to trust this certificate myself (imagine a self-issued certificate where I can validate fingerprint etc.): how can I make a request and validate/trust the certificate?
I probably need to use openssl to download the certificate, load it into my file and fill tls.Config struct !?
Security note: Disabling security checks is dangerous and should be avoided
You can disable security checks globally for all requests of the default client:
package main
import (
"fmt"
"net/http"
"crypto/tls"
)
func main() {
http.DefaultTransport.(*http.Transport).TLSClientConfig = &tls.Config{InsecureSkipVerify: true}
_, err := http.Get("https://golang.org/")
if err != nil {
fmt.Println(err)
}
}
You can disable security check for a client:
package main
import (
"fmt"
"net/http"
"crypto/tls"
)
func main() {
tr := &http.Transport{
TLSClientConfig: &tls.Config{InsecureSkipVerify: true},
}
client := &http.Client{Transport: tr}
_, err := client.Get("https://golang.org/")
if err != nil {
fmt.Println(err)
}
}
Proper way (as of Go 1.13) (provided by answer below):
customTransport := http.DefaultTransport.(*http.Transport).Clone()
customTransport.TLSClientConfig = &tls.Config{InsecureSkipVerify: true}
client := &http.Client{Transport: customTransport}
Original Answer:
Here's a way to do it without losing the default settings of the DefaultTransport, and without needing the fake request as per user comment.
defaultTransport := http.DefaultTransport.(*http.Transport)
// Create new Transport that ignores self-signed SSL
customTransport := &http.Transport{
Proxy: defaultTransport.Proxy,
DialContext: defaultTransport.DialContext,
MaxIdleConns: defaultTransport.MaxIdleConns,
IdleConnTimeout: defaultTransport.IdleConnTimeout,
ExpectContinueTimeout: defaultTransport.ExpectContinueTimeout,
TLSHandshakeTimeout: defaultTransport.TLSHandshakeTimeout,
TLSClientConfig: &tls.Config{InsecureSkipVerify: true},
}
client := &http.Client{Transport: customTransport}
Shorter way:
customTransport := &(*http.DefaultTransport.(*http.Transport)) // make shallow copy
customTransport.TLSClientConfig = &tls.Config{InsecureSkipVerify: true}
client := &http.Client{Transport: customTransport}
Warning: For testing/development purposes only. Anything else, proceed at your own risk!!!
All of these answers are wrong! Do not use InsecureSkipVerify to deal with a CN that doesn't match the hostname. The Go developers unwisely were adamant about not disabling hostname checks (which has legitimate uses - tunnels, nats, shared cluster certs, etc), while also having something that looks similar but actually completely ignores the certificate check. You need to know that the certificate is valid and signed by a cert that you trust. But in common scenarios, you know that the CN won't match the hostname you connected with. For those, set ServerName on tls.Config. If tls.Config.ServerName == remoteServerCN, then the certificate check will succeed. This is what you want. InsecureSkipVerify means that there is NO authentication; and it's ripe for a Man-In-The-Middle; defeating the purpose of using TLS.
There is one legitimate use for InsecureSkipVerify: use it to connect to a host and grab its certificate, then immediately disconnect. If you setup your code to use InsecureSkipVerify, it's generally because you didn't set ServerName properly (it will need to come from an env var or something - don't belly-ache about this requirement... do it correctly).
In particular, if you use client certs and rely on them for authentication, you basically have a fake login that doesn't actually login any more. Refuse code that does InsecureSkipVerify, or you will learn what is wrong with it the hard way!
The correct way to do this if you want to maintain the default transport settings is now (as of Go 1.13):
customTransport := http.DefaultTransport.(*http.Transport).Clone()
customTransport.TLSClientConfig = &tls.Config{InsecureSkipVerify: true}
client = &http.Client{Transport: customTransport}
Transport.Clone makes a deep copy of the transport. This way you don't have to worry about missing any new fields that get added to the Transport struct over time.
If you want to use the default settings from http package, so you don't need to create a new Transport and Client object, you can change to ignore the certificate verification like this:
tr := http.DefaultTransport.(*http.Transport)
tr.TLSClientConfig.InsecureSkipVerify = true
Generally, The DNS Domain of the URL MUST match the Certificate Subject of the certificate.
In former times this could be either by setting the domain as cn of the certificate or by having the domain set as a Subject Alternative Name.
Support for cn was deprecated for a long time (since 2000 in RFC 2818) and Chrome browser will not even look at the cn anymore so today you need to have the DNS Domain of the URL as a Subject Alternative Name.
RFC 6125 which forbids checking the cn if SAN for DNS Domain is present, but not if SAN for IP Address is present. RFC 6125 also repeats that cn is deprecated which was already said in RFC 2818. And the Certification Authority Browser Forum to be present which in combination with RFC 6125 essentially means that cn will never be checked for DNS Domain name.