I find myself needing to set up a WebSocket connection in a hostile environment in which a firewall sniffs SNI information from TLS which I'd rather it didn't. In my particular case, the WebSocket server does not use SNI for request handling, so as such, the SNI part of the handshake could be safely removed.
My question then becomes: In the golang.org WebSocket package, golang.org/x/net/websocket, what is the simplest way to strip SNI information while retaining validation of the provided chain?
The best I have been able to come up with is to simply replace the hostname of the URL to be dialled with its corresponding IP. This causes crypto/tls to never add the problematic SNI information, but, in the solution I was able to come up with, a custom validator ends up having to be provided to validate the chain:
func dial(url string, origin string) (*websocket.Conn, error) {
// Use system resolver to get IP of host
hostRegExp := regexp.MustCompile("//([^/]+)/")
host := hostRegExp.FindStringSubmatch(url)[1]
addrs, err := net.LookupHost(host)
if err != nil {
return fmt.Errorf("Could not resolve address of %s: %v", host, err)
}
ip := addrs[0]
// Replace the hostname in the given URL with its IP instead
newURL := strings.Replace(url, host, ip, 1)
config, _ := websocket.NewConfig(newURL, origin)
// As we have removed the hostname, the Go TLS package will not know what to
// validate the certificate DNS names against, so we have to provide a custom
// verifier based on the hostname we threw away.
config.TlsConfig = &tls.Config{
InsecureSkipVerify: true,
VerifyPeerCertificate: verifier(host),
}
return websocket.DialConfig(config)
}
func verifier(host string) func(rawCerts [][]byte, verifiedChains [][]*x509.Certificate) error {
return func(rawCerts [][]byte, verifiedChains [][]*x509.Certificate) error {
// For simplicity, let us only consider the case in which the first certificate is the one
// to validate, and in which it is signed directly by a CA, with no parsing of
// intermediate certificates required.
opts := x509.VerifyOptions{
DNSName: host,
}
rawCert := rawCerts[0]
cert, err := x509.ParseCertificate(rawCert)
if err != nil {
return err
}
_, err = cert.Verify(opts)
return err
}
}
This totally works but seems rather clunky. Is there a simpler approach? (Ideally one that is not specific to WebSocket applications but works for TLS in general; the exact same idea as above could be applied to HTTPS.)
Related
I am trying to understand the mutual TLS working, I have the following example:
I have a client who wants to connect to server "svc1.example.com"
but the server has a
server certificate with a commonName as "svc1.example.cloud" and a SAN
as "svc.example.test.cloud".
Now when I make a GET request, I get the following:
x509: certificate is valid for svc.example.test.cloud, not svc1.example.com.
So, my question is should I make a the TLS clientConfig changes to include the servername? or should I add a custom verifyPeerCertificate function in the TLS client config, something like below?
Please, let me know, what should be the Servername and what should I check for in the verifyPeerCertificate function.
func customverify(customCName func(*x509.Certificate) bool) func(rawCerts [][]byte, verifiedChains [][]*x509.Certificate) error {
if customCName == nil {
return nil
}
return func(_ [][]byte, verifiedChains [][]*x509.Certificate) error {
for _, certs := range verifiedChains {
leaf := certs[0]
if customCName(leaf) {
return nil
}
}
return fmt.Errorf("client identity verification failed")
}
}
func configureClient(certFile, keyFile string) (*http.Client, error) {
certpool, err := addRootCA()
if err != nil {
return nil, err
}
cert, err := tls.LoadX509KeyPair(certFile, keyFile)
if err != nil {
return nil, err
}
transport := ytls.NewClientTransport()
transport.TLSClientConfig.Certificates = []tls.Certificate{cert}
transport.TLSClientConfig.RootCAs = certpool
//transport.TLSClientConfig.ServerName = expectedCName
transport.TLSClientConfig.VerifyPeerCertificate = customverify(func(cert *x509.Certificate) bool {
return cert.Subject.CommonName == "svc1.example.cloud"
})
httpClient := &http.Client{Transport: transport}
return httpClient, nil
}
Since x509: certificate is valid for svc.example.test.cloud, so transport.TLSClientConfig.ServerName = "svc.example.test.cloud"
From https://golang.org/pkg/crypto/tls/#Config
VerifyPeerCertificate, if not nil, is called after normal
certificate verification by either a TLS client or server. It
receives the raw ASN.1 certificates provided by the peer and also
any verified chains that normal processing found. If it returns a
non-nil error, the handshake is aborted and that error results.
If normal verification fails then the handshake will abort before
considering this callback. If normal verification is disabled by
setting InsecureSkipVerify, or (for a server) when ClientAuth is
RequestClientCert or RequireAnyClientCert, then this callback will
be considered but the verifiedChains argument will always be nil.
VerifyPeerCertificate func(rawCerts [][]byte, verifiedChains [][]*x509.Certificate) error
So if normal verification fails, then VerifyPeerCertificate won't get called. Also if normal verification is passed, i don't think you need this extra check VerifyPeerCertificate.
I'm starting to learn golang and I'm trying to make a simple http client that will get a list of virtual machines from one of our oVirt clusters. The API that I'm trying to access has a self-signed certificate (auto generated during the cluster installation) and golang's http.client encounters a problem when serializing the time from the certificate. Below you can find the code and the output.
package main
import (
"fmt"
"io/ioutil"
"net/http"
"crypto/tls"
)
func do_request(url string) ([]byte, error) {
// ignore self signed certificates
transCfg := &http.Transport{
TLSClientConfig: &tls.Config {
InsecureSkipVerify: true,
},
}
// http client
client := &http.Client{Transport: transCfg}
// request with basic auth
req, _ := http.NewRequest("GET", url, nil)
req.SetBasicAuth("user","pass")
resp, err := client.Do(req)
// error?
if err != nil {
fmt.Printf("Error : %s", err)
return nil, err
}
defer resp.Body.Close()
body, _ := ioutil.ReadAll(resp.Body)
return []byte(body), nil
}
func main() {
body, _ := do_request("https://ovirt.example.com/")
fmt.Println("response Status:", string(body))
}
and the error when I'm trying to compile:
$ go run http-get.go
Error : Get https://ovirt.example.com/: tls: failed to parse certificate from server: asn1: time did not serialize back to the original value and may be invalid: given "141020123326+0000", but serialized as "141020123326Z"response Status:
Is there any way to ignore this verification? I tried making a request using other programming languages (python, ruby) and skipping insecure certificates seems to be enough.
Thank you!
PS: I know the proper solution is to change the certificate with a valid one, but for the moment I cannot do this.
Unfortunately, you've encountered an error that you cannot get around in Go. This is buried deep in the cypto/x509 and encoding/asn1 packages without a way to ignore. Specifically, asn1.parseUTCTime is expecting the time format to be "0601021504Z0700", but your server is sending "0601021504+0000". Technically, that is a known format but encoding/asn1 does not support it.
There are only 2 solutions that I can come up with that do not require a code change for golang.
1) Edit the encoding/asn1 package in your go src directory and then rebuild all the standard packages with go build -a
2) Create your own customer tls, x509 and asn1 packages to use the format your server is sending.
Hope this helps.
P.S. I've opened an issue with the Go developers to see if it can resolved by them at some later point Issue Link
Possible ASN1 UtcTime Formats.
I'm trying to write tests for a package that makes requests to a web service. I'm running into issues probably due to my lack of understanding of TLS.
Currently my test looks something like this:
func TestSimple() {
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(200)
fmt.Fprintf(w, `{ "fake" : "json data here" }`)
}))
transport := &http.Transport{
Proxy: func(req *http.Request) (*url.URL, error) {
return url.Parse(server.URL)
},
}
// Client is the type in my package that makes requests
client := Client{
c: http.Client{Transport: transport},
}
client.DoRequest() // ...
}
My package has a package variable (I'd like for it to be a constant..) for the base address of the web service to query. It is an https URL. The test server I created above is plain HTTP, no TLS.
By default, my test fails with the error "tls: first record does not look like a TLS handshake."
To get this to work, my tests change the package variable to a plain http URL instead of https before making the query.
Is there any way around this? Can I make the package variable a constant (https), and either set up a http.Transport that "downgrades" to unencrypted HTTP, or use httptest.NewTLSServer() instead?
(When I try to use NewTLSServer() I get "http: TLS handshake error from 127.0.0.1:45678: tls: oversized record received with length 20037")
Most of the behavior in net/http can be mocked, extended, or altered. Although http.Client is a concrete type that implements HTTP client semantics, all of its fields are exported and may be customized.
The Client.Transport field, in particular, may be replaced to make the Client do anything from using custom protocols (such as ftp:// or file://) to connecting directly to local handlers (without generating HTTP protocol bytes or sending anything over the network).
The client functions, such as http.Get, all utilize the exported http.DefaultClient package variable (which you may modify), so code that utilizes these convenience functions does not, for example, have to be changed to call methods on a custom Client variable. Note that while it would be unreasonable to modify global behavior in a publicly-available library, it's very useful to do so in applications and tests (including library tests).
http://play.golang.org/p/afljO086iB contains a custom http.RoundTripper that rewrites the request URL so that it'll be routed to a locally hosted httptest.Server, and another example that directly passes the request to an http.Handler, along with a custom http.ResponseWriter implementation, in order to create an http.Response. The second approach isn't as diligent as the first (it doesn't fill out as many fields in the Response value) but is more efficient, and should be compatible enough to work with most handlers and client callers.
The above-linked code is included below as well:
package main
import (
"fmt"
"io"
"log"
"net/http"
"net/http/httptest"
"net/url"
"os"
"path"
"strings"
)
func Handler(w http.ResponseWriter, r *http.Request) {
fmt.Fprintf(w, "hello %s\n", path.Base(r.URL.Path))
}
func main() {
s := httptest.NewServer(http.HandlerFunc(Handler))
u, err := url.Parse(s.URL)
if err != nil {
log.Fatalln("failed to parse httptest.Server URL:", err)
}
http.DefaultClient.Transport = RewriteTransport{URL: u}
resp, err := http.Get("https://google.com/path-one")
if err != nil {
log.Fatalln("failed to send first request:", err)
}
fmt.Println("[First Response]")
resp.Write(os.Stdout)
fmt.Print("\n", strings.Repeat("-", 80), "\n\n")
http.DefaultClient.Transport = HandlerTransport{http.HandlerFunc(Handler)}
resp, err = http.Get("https://google.com/path-two")
if err != nil {
log.Fatalln("failed to send second request:", err)
}
fmt.Println("[Second Response]")
resp.Write(os.Stdout)
}
// RewriteTransport is an http.RoundTripper that rewrites requests
// using the provided URL's Scheme and Host, and its Path as a prefix.
// The Opaque field is untouched.
// If Transport is nil, http.DefaultTransport is used
type RewriteTransport struct {
Transport http.RoundTripper
URL *url.URL
}
func (t RewriteTransport) RoundTrip(req *http.Request) (*http.Response, error) {
// note that url.URL.ResolveReference doesn't work here
// since t.u is an absolute url
req.URL.Scheme = t.URL.Scheme
req.URL.Host = t.URL.Host
req.URL.Path = path.Join(t.URL.Path, req.URL.Path)
rt := t.Transport
if rt == nil {
rt = http.DefaultTransport
}
return rt.RoundTrip(req)
}
type HandlerTransport struct{ h http.Handler }
func (t HandlerTransport) RoundTrip(req *http.Request) (*http.Response, error) {
r, w := io.Pipe()
resp := &http.Response{
Proto: "HTTP/1.1",
ProtoMajor: 1,
ProtoMinor: 1,
Header: make(http.Header),
Body: r,
Request: req,
}
ready := make(chan struct{})
prw := &pipeResponseWriter{r, w, resp, ready}
go func() {
defer w.Close()
t.h.ServeHTTP(prw, req)
}()
<-ready
return resp, nil
}
type pipeResponseWriter struct {
r *io.PipeReader
w *io.PipeWriter
resp *http.Response
ready chan<- struct{}
}
func (w *pipeResponseWriter) Header() http.Header {
return w.resp.Header
}
func (w *pipeResponseWriter) Write(p []byte) (int, error) {
if w.ready != nil {
w.WriteHeader(http.StatusOK)
}
return w.w.Write(p)
}
func (w *pipeResponseWriter) WriteHeader(status int) {
if w.ready == nil {
// already called
return
}
w.resp.StatusCode = status
w.resp.Status = fmt.Sprintf("%d %s", status, http.StatusText(status))
close(w.ready)
w.ready = nil
}
The reason you're getting the error http: TLS handshake error from 127.0.0.1:45678: tls: oversized record received with length 20037 is because https requires a domain name (not an IP Address). Domain names are SSL certificates are assigned to.
Start the httptest server in TLS mode with your own certs
cert, err := tls.LoadX509KeyPair("cert.pem", "key.pem")
if err != nil {
log.Panic("bad server certs: ", err)
}
certs := []tls.Certificate{cert}
server = httptest.NewUnstartedServer(router)
server.TLS = &tls.Config{Certificates: certs}
server.StartTLS()
serverPort = ":" + strings.Split(server.URL, ":")[2] // it's always https://127.0.0.1:<port>
server.URL = "https://sub.domain.com" + serverPort
To provide a valid SSL certificate for a connection are the options of:
Not supplying a cert and key
Supplying a self-signed cert and key
Supplying a real valid cert and key
No Cert
If you don't supply your own cert, then an example.com cert is loaded as default.
Self-Signed Cert
To create a testing cert can use the included self-signed cert generator at $GOROOT/src/crypto/tls/generate_cert.go --host "*.domain.name"
You'll get x509: certificate signed by unknown authority warnings because it's self-signed so you'll need to have your client skip those warnings, by adding the following to your http.Transport field:
TLSClientConfig: &tls.Config{InsecureSkipVerify: true}
Valid Real Cert
Finally, if you're going to use a real cert, then save the valid cert and key where they can be loaded.
The key here is to use server.URL = https://sub.domain.com to supply your own domain.
From Go 1.9+ you can use func (s *Server) Client() *http.Client in the httptest package:
Client returns an HTTP client configured for making requests to the server. It is configured to trust the server's TLS test certificate and will close its idle connections on Server.Close.
Example from the package:
package main
import (
"fmt"
"io"
"log"
"net/http"
"net/http/httptest"
)
func main() {
ts := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
fmt.Fprintln(w, "Hello, client")
}))
defer ts.Close()
client := ts.Client()
res, err := client.Get(ts.URL)
if err != nil {
log.Fatal(err)
}
greeting, err := io.ReadAll(res.Body)
res.Body.Close()
if err != nil {
log.Fatal(err)
}
fmt.Printf("%s", greeting)
}
How can I check the fingerprints of the server SSL/TLS certificates during a http request in golang?
This ruby code shows what I want to do in Go:
#verify_callback = proc do |preverify_ok, store_context|
if preverify_ok and store_context.error == 0
certificate = OpenSSL::X509::Certificate.new(store_context.chain[0])
fingerprint = Digest::SHA1.hexdigest(certificate.to_der).upcase.scan(/../).join(":")
$valid_fingerprints.include?(fingerprint)
else
false
end
end
In general the process of generating a certificate fingerprint in Go is pretty simple. If you already have an x509.Certificate struct, stored in cert, all you need to do is
sha1Fingerprint := sha1.Sum(cert.Raw)
Getting certificates from an HTTP response struct after the request is complete is also pretty easy (use resp.TLS.PeerCertificates), but it doesn't seem like that's what you need.
If you need access to the server's certificate at TLS connection set up time, I think you'll need to create your own http.Transport and hand it a custom implementation of DialTLS. You'd then use that transport when configuring an http.Client to make your outbound requests.
Within your custom DialTLS func you'd have access to connection state information like the server's certificate chain, and you could perform the SHA1 fingerprint generation from there.
You probably shouldn't implement certificate checking yourself, but let the net/http do the checking based on the valid CAs you provide. Also, usually working directly with fingerprints isn't worth the trouble.
For example, this is how you set up a HTTPS server that requires clients to authenticate by using a certificate. The client certificate must be signed by the CA, or the SSL/TLS handshake stops.
// Server's own certificate & key
cert, err := tls.LoadX509KeyPair("server.crt", "server.key")
if err != nil {
panic(err)
}
// Load the CA certificate(s)
capool := x509.NewCertPool()
cacert, err := ioutil.ReadFile("ca.crt")
if err != nil {
panic(err)
}
capool.AppendCertsFromPEM(cacert)
// Server configuration
config := tls.Config{Certificates: []tls.Certificate{cert}, ClientCAs: capool, ClientAuth: tls.RequireAndVerifyClientCert}
config.NextProtos = []string{"http/1.1"}
config.Rand = rand.Reader // Strictly not necessary, should be default
// TLS web server
myTLSWebServer := &http.Server{Addr: "myaddress", TLSConfig: &config, Handler: nil}
// .. proceed with setting handlers etc
http.HandleFunc("/", myHandler)
// Bind to port and start the server up
conn, err := net.Listen("tcp", settings.ServiceAddress)
if err != nil {
panic(err)
}
tlsListener := tls.NewListener(conn, &config)
myTLSWebServer.Serve(tlsListener)
Reading the documentation for tls.Config will show you that by changing the parameters (ClientAuth, ClientCAs, Certificates, RootCAs) you can easily select different modes for checking the certificates. You usually get failures returned in error.
If you really insist on checking fingerprints, you can retrieve the TLS status from Request TLS *tls.ConnectionState. I think you should probably use the Signature from that struct for fingerprinting.. Off the top of my head, something roughly along the lines of
func lol(r *http.Request) {
tls := r.TLS
if tls != nil {
// Try the first one for simplicity
cert := tls.PeerCertificates[0]
signature := cert.Signature
// Do something with the signature
}
}
should do the trick.
I have the following certificate hierarchy:
Root-->CA-->3 leaf certificates
The entire chain has both serverAuth and clientAuth as extended key usages explicitly defined.
In my go code, I create a tls.Config object like so:
func parseCert(certFile, keyFile string) (cert tls.Certificate, err error) {
certPEMBlock , err := ioutil.ReadFile(certFile)
if err != nil {
return
}
var certDERBlock *pem.Block
for {
certDERBlock, certPEMBlock = pem.Decode(certPEMBlock)
if certDERBlock == nil {
break
}
if certDERBlock.Type == "CERTIFICATE" {
cert.Certificate = append(cert.Certificate, certDERBlock.Bytes)
}
}
// Need to flip the array because openssl gives it to us in the opposite format than golang tls expects.
cpy := make([][]byte, len(cert.Certificate))
copy(cpy, cert.Certificate)
var j = 0
for i := len(cpy)-1; i >=0; i-- {
cert.Certificate[j] = cert.Certificate[i]
j++
}
keyData, err := ioutil.ReadFile(keyFile)
if err != nil {
return
}
block, _ := pem.Decode(keyData)
if err != nil {
return
}
ecdsaKey, err := x509.ParseECPrivateKey(block.Bytes)
if err != nil {
return
}
cert.PrivateKey = ecdsaKey
return
}
// configure and create a tls.Config instance using the provided cert, key, and ca cert files.
func configureTLS(certFile, keyFile, caCertFile string) (tlsConfig *tls.Config, err error) {
c, err := parseCert(certFile, keyFile)
if err != nil {
return
}
ciphers := []uint16 {
tls.TLS_RSA_WITH_AES_256_CBC_SHA,
tls.TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,
}
certPool := x509.NewCertPool()
buf, err := ioutil.ReadFile(caCertFile)
if nil != err {
log.Println("failed to load ca cert")
log.Fatal(seelog.Errorf("failed to load ca cert.\n%s", err))
}
if !certPool.AppendCertsFromPEM(buf) {
log.Fatalln("Failed to parse truststore")
}
tlsConfig = &tls.Config {
CipherSuites: ciphers,
ClientAuth: tls.RequireAndVerifyClientCert,
PreferServerCipherSuites: true,
RootCAs: certPool,
ClientCAs: certPool,
Certificates: []tls.Certificate{c},
}
return
}
certFile is the certificate chain file and keyFile is the private key file. caCertFile is the truststore and consists of just the root certificate
So basically, here is what I expect to have inside of my tls.Config object that comes out of this function:
RootCAs: Just the root certificate from caCertFile
ClientCAs: Again, just the root certificate from caCertFile, same as RootCAs
Certificates: A single certificate chain, containing all of the certificates in certFile, ordered to be leaf first.
Now, I have 3 pieces here. A server, a relay, and a client. The client connects directly to the relay, which in turn forwards the request to the server. All three pieces use the same configuration code, of course using different certs/keys. The caCertFile is the same between all 3 pieces.
Now, if I stand up the server and the relay and connect to the relay from my browser, all goes well, so I can assume that the connection between relay and server is fine. The issue comes about when I try to connect my client to the relay. When I do so, the TLS handshake fails and the following error is returned:
x509: certificate signed by unknown authority
On the relay side of things, I get the following error:
http: TLS handshake error from : remote error: bad certificate
I am really at a loss here. I obviously have something setup incorrectly, but I am not sure what. It's really weird that it works from the browser (meaning that the config is correct from relay to server), but it doesn't work with the same config from my client.
Update:
So if I add InsecureSkipVerify: true to my tls.Config object on both the relay and the client, the errors change to:
on the client: remote error: bad certificate
and on the relay: http: TLS handshake error from : tls: client didn't provide a certificate
So it looks like the client is rejecting the certificate on from the server (the relay) due to it being invalid for some reason and thus never sending its certificate to the server (the relay).
I really wish go had better logging. I can't even hook into this process to see what, exactly, is going on.
When you say
Need to flip the array because openssl gives it to us in the opposite format than golang tls expects.
I have used certificates generated by openssl and had no problem opening them with:
tls.LoadX509KeyPair(cert, key)
Anyway, the error message bad certificate is due to the server not managing to match the client-provided certificate against its RootCAs. I have also had this problem in Go using self-signed certificats and the only work-around I've found is to install the caCertFile into the machines system certs, and use x509.SystemCertPool() instead of x509.NewCertPool().
Maybe someone else will have another solution?
Beside what beldin0 suggested.
I have tried another way to do this.
caCertPool := x509.NewCertPool()
caCertPool.AppendCertsFromPEM(crt)
client := &http.Client{
//some config
Transport: &http.Transport{
TLSClientConfig: &tls.Config{
RootCAs: caCertPool,
},
},
}
Here, the variable "crt" is the content in your certificate.
Basically, you just add it into your code(or read as a config file).
Then everything would be fine.