Our server access internet through a proxy. When I try to run a pull command such as
sudo docker run -t -i ubuntu:14.04 /bin/bash
I get the below error:
Get https://index.docker.io/v1/repositories/ubuntu/images: tls: failed to parse
certificate from server: x509: negative serial number
The wget command wget -S -d -O - https://get.docker.io yields the below output:
Setting --output-document (outputdocument) to - DEBUG output created
by Wget 1.13.4 on linux-gnu.
URI encoding = UTF-8' URI encoding =UTF-8'
--2014-08-27 17:13:46-- https://get.docker.io/ Connecting to :... connected. Created socket 3. Releasing
0x00000000016829f0 (new refcount 0). Deleting unused
0x00000000016829f0.
---request begin--- CONNECT get.docker.io:443 HTTP/1.1 User-Agent: Wget/1.13.4 (linux-gnu) Proxy-Authorization: Basic
Y3RzXDMxMzMwMDpzd2VldGZlbC4yOQ==
---request end--- proxy responded with: [HTTP/1.1 200 Connection established Date: Wed, 27 Aug 2014 11:49:52 GMT Age: 0 Via: 1.0
xaahshshhds
] Initiating SSL handshake. Handshake successful; connected socket 3
to SSL handle 0x00000000016831c0 certificate: subject:
/emailAddress=aaa#bbbb.com/C=yy/ST=aa/L=xx/O=yy/OU=mycompany/CN=get.docker.io
issuer:
/emailAddress=aaa#bbbb.com/C=yy/ST=aa/L=xx/O=yy/OU=mycompany/CN=mycompany
ERROR: cannot verify get.docker.io's certificate, issued by
/emailAddress=aaa#bbbb.com/C=yy/ST=aa/L=xx/O=yy/OU=mycompany/CN=mycompany':
Unable to locally verify the issuer's authority. To connect to
get.docker.io insecurely, use--no-check-certificate'. Closed 3/SSL
0x00000000016831c0
Please give me some directions on how I should go about this issue.
EDIT:
I ve now disabled the proxy for this IP segment but I still get the same error.
The command: wget -S -d -O - https://get.docker.io gets the below output now:
Setting --output-document (outputdocument) to -
DEBUG output created by Wget 1.13.4 on linux-gnu.
URI encoding = `UTF-8'
--2014-09-04 11:26:12-- https://get.docker.io/
Resolving get.docker.io (get.docker.io)... 162.242.195.77
Caching get.docker.io => 162.242.195.77
Connecting to get.docker.io (get.docker.io)|162.242.195.77|:443... connected.
Created socket 3.
Releasing 0x00000000022d8fd0 (new refcount 1).
Initiating SSL handshake.
Handshake successful; connected socket 3 to SSL handle 0x00000000022dabd0
certificate:
subject: /serialNumber=exkd9EjUozUulWIyUDurQPMEPBLSc2Bq/OU=GT98568428/OU=See www.rapidssl.com/resources/cps (c)13/OU=Domain Control Validated - RapidSSL(R)/CN=*.docker.io
issuer: /C=US/O=GeoTrust, Inc./CN=RapidSSL CA
X509 certificate successfully verified and matches host get.docker.io
---request begin---
GET / HTTP/1.1
User-Agent: Wget/1.13.4 (linux-gnu)
Accept: */*
Host: get.docker.io
Connection: Keep-Alive
---request end---
HTTP request sent, awaiting response...
---response begin---
HTTP/1.1 503 Service Unavailable
Server: nginx/1.7.1
Date: Thu, 04 Sep 2014 06:03:28 GMT
Content-Type: text/html
Transfer-Encoding: chunked
Connection: keep-alive
Cache-Control: no-cache
---response end---
HTTP/1.1 503 Service Unavailable
Server: nginx/1.7.1
Date: Thu, 04 Sep 2014 06:03:28 GMT
Content-Type: text/html
Transfer-Encoding: chunked
Connection: keep-alive
Cache-Control: no-cache
Registered socket 3 for persistent reuse.
Skipping 108 bytes of body: [<html><body><h1>503 Service Unavailable</h1>
No server is available to handle this request.
</body></html>
] done.
2014-09-04 11:26:13 ERROR 503: Service Unavailable.
subject: /emailAddress=aaa#bbbb.com/C=yy/ST=aa/L=xx/O=yy/OU=mycompany/CN=get.docker.io
issuer: /emailAddress=aaa#bbbb.com/C=yy/ST=aa/L=xx/O=yy/OU=mycompany/CN=mycompany
It looks like the proxy in your company uses SSL interception to inspect SSL traffic, which means that you get a certificate signed by the proxy CA of your company instead of the original certificate. It also looks like that this proxy CA is not trusted by your system and thus the verification fails.
I would recommend that you contact your firewall administrator on how to deal with the problem. Either they will add an exception for the SSL inspection, or they will tell you which certificate you need to import as trusted in your system.
This should be fixed for any Docker compiled with Go 1.6+, see: https://github.com/golang/go/commit/a0ea93dea5f5741addc8c96b7ed037d0e359e33f.
Related
I'm installing OpenVPN Access Server on a Google Cloud instance. Its webUI listens on port 943 using https. It has a self-signed certificate whose name doesn't match the server's hostname (10.150.0.2). I can't start an SSH tunnel. I'm looking for a way to troubleshoot the connection from the IAP service to my server.
The command I'm running is gcloud compute start-iap-tunnel vpn 943 --local-host-port=localhost:943 I receive the normal Testing if tunnel connection works message.
It errs out with ERROR: (gcloud.compute.start-iap-tunnel) While checking if a connection can be made: Error while connecting [4003: 'failed to connect to backend']. (Failed to connect to port 943)
If I add --log-http to the command invocation the relevant information follows (it looks like a normal req/resp cycle with a 200 that I assume is from my client to the IAP service):
Testing if tunnel connection works.
=======================
==== request start ====
uri: https://oauth2.googleapis.com/token
method: POST
== headers start ==
b'content-type': b'application/x-www-form-urlencoded'
b'user-agent': b'google-cloud-sdk gcloud/367.0.0 command/gcloud.compute.start-iap-tunnel invocation-id/db27de82264f47fcb63f6680afaa8327 environment/None environment-version/None interactive/False from-script/False python/3.7.9 term/xterm-256color (Macintosh; Intel Mac OS X 21.2.0)'
== headers end ==
== body start ==
Body redacted: Contains oauth token. Set log_http_redact_token property to false to print the body of this request.
== body end ==
==== request end ====
---- response start ----
status: 200
-- headers start --
Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
Content-Encoding: gzip
Content-Type: application/json; charset=utf-8
Date: Fri, 24 Dec 2021 02:11:52 GMT
Expires: Mon, 01 Jan 1990 00:00:00 GMT
Pragma: no-cache
Server: scaffolding on HTTPServer2
Transfer-Encoding: chunked
Vary: Origin, X-Origin, Referer
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 0
-- headers end --
-- body start --
Body redacted: Contains oauth token. Set log_http_redact_token property to false to print the body of this response.
-- body end --
total round trip time (request+response): 0.246 secs
---- response end ----
----------------------
ERROR: (gcloud.compute.start-iap-tunnel) While checking if a connection can be made: Error while connecting [4003: 'failed to connect to backend']. (Failed to connect to port 943)
To my knowledge this is the limit of easily accessible troubleshooting for start-tap-tunnel.
Moving on to the local machine we can connect to 10.150.0.2:943 before puking a la certificate.
root#viongier:/usr/local/openvpn_as# wget https://10.150.0.2:943
--2021-12-24 02:01:47-- https://10.150.0.2:943/
Connecting to 10.150.0.2:943... connected.
ERROR: The certificate of ‘10.150.0.2’ is not trusted.
ERROR: The certificate of ‘10.150.0.2’ doesn't have a known issuer.
The certificate's owner does not match hostname ‘10.150.0.2’
It seems to me that my client happily connects to the IAP service which fails to connect to my server. I would expect to see an IAP error if it was erring out because of the cert. The only thing I can think of to test this is by generating a certificate whose issuer google likes. (LetsEncrypt for example.)
This message means that the backend does not have a socket open in the listening state. Common reasons are that no service has been started or a firewall is blocking the port.
To allow the Identity Aware Proxy into your VPC, allow traffic from 35.235.240.0/20.
ERROR: (gcloud.compute.start-iap-tunnel) While checking if a
connection can be made: Error while connecting [4003: 'failed to
connect to backend']. (Failed to connect to port 943)
This error means that the certificate provided does not match the address that the connection is made to:
ERROR: The certificate of ‘10.150.0.2’ is not trusted. ERROR: The
certificate of ‘10.150.0.2’ doesn't have a known issuer. The
certificate's owner does not match hostname ‘10.150.0.2’
Some clients, such as wget support ignoring SSL certificate validation. For wget see the --no-check-certificate flag.
Once you solve that problem you will run into another set of problems:
Under normal circumstances, you can not use HTTPS with tunnels. Tunnels are a form of man in the middle. There are tricks that can be employed, none of them secure.
Commercial SSL certificates do not support IP addresses only public domain names. You would need to create your own self-signed certificate, which would not be trusted or do not validate the certificate.
The last issue is that HTTPS endpoints require encryption negotiation from the client party. The start-iap-tunnel command does not initiate encryption (TLS negotiation). This command also does not do any form of certificate exchange and that is why you do not see an IAP error about certificates. This command only transfers data between the tunnel endpoints.
In summary, you cannot use HTTPS with TCP / SSH tunnels without deploying tricks and/or disabling features which defeats the purpose of HTTPS.
Allow IAP traffic through the firewall allowed my external client to connect to the internal port 943 via an IAP tunnel.
Allowing port 943 from 35.235.240.0/20 solved my problem.
More information is available at the GCP IAP docs
I have a simple app where the client is sending a request to join a websocket channel upon page load via a wss connection. I have deployed my app to heroku at domain my_app.heroku.com. Whenever I open my site, I get the error Error in connection establishment: net::ERR_SSL_PROTOCOL_ERROR. I understand that this is an SSL issue, and also that to use Heroku's SSL, you must be on a hobbyist account. So, I upgraded the app to hobbyist (one day ago), but it has not fixed the problem.
I have googled the issue, and from what I see it appears that additional setup beyond upgrading to Heroku Hobby should only be required if using a custom domain-name, which I am not. How can I get the SSL working on my my_app.heroku.com domain with the Hobby dyno?
EDIT: If I run curl -vI <my_app>.heroku_app.com I get the following:
curl -vI https://<my_app>.herokuapp.com
* Rebuilt URL to: https://<my_app>.herokuapp.com/
* Trying __.___.___.___...
* Connected to <my_app>.herokuapp.com (__.___.___.___) port 443
(#0)
* found 148 certificates in /etc/ssl/certs/ca-certificates.crt
* found 592 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* SSL connection using TLS1.2 / ECDHE_RSA_AES_128_GCM_SHA256
* server certificate verification OK
* server certificate status verification SKIPPED
* common name: *.herokuapp.com (matched)
* server certificate expiration date OK
* server certificate activation date OK
* certificate public key: RSA
* certificate version: #3
* subject: C=US,ST=California,L=San Francisco,O=Heroku\, Inc.,CN=*.herokuapp.com
* start date: Wed, 19 Apr 2017 00:00:00 GMT
* expire date: Mon, 22 Jun 2020 12:00:00 GMT
* issuer: C=US,O=DigiCert Inc,OU=www.digicert.com,CN=DigiCert SHA2
High Assurance Server CA
* compression: NULL
* ALPN, server did not agree to a protocol
> HEAD / HTTP/1.1
> Host: <my_app>.herokuapp.com
> User-Agent: curl/7.47.0
> Accept: */*
>
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< Server: Cowboy
Server: Cowboy
< Connection: keep-alive
Connection: keep-alive
< Content-Type: text/html; charset=utf-8
Content-Type: text/html; charset=utf-8
< Content-Length: 1154
Content-Length: 1154
< Date: Thu, 22 Nov 2018 08:39:51 GMT
Date: Thu, 22 Nov 2018 08:39:51 GMT
< Via: 1.1 vegur
Via: 1.1 vegur
I found the answer after finally finding this question. I thought all along the issue was due to setting up SSL certificates on Heroku, however it was due to my client side code. Per the tutorial I was following, the WS connection was "wss://my_app.herokuapp.com:80/subscribe". Removing the port number was the fix.
As discussed in my other question there is no support for websockets authentication in Knox, but as a temporary solution we could handle authentication in our backend service. Our test has shown however that Knox does not pass Authorization header to the backend.
[client]$ curl -i -u '<user>:<password>' https://knox-server/gateway/default/myservice/ping
# 8090 is our backend port
[knox-server]$ ngrep -W byline port 8090
interface: eth0
filter: ( port 8090 ) and ((ip || ip6) || (vlan && (ip || ip6)))
#
T <knox-server>:59118 -> <myservice>:8090 [AP]
GET /ping?doAs=<user> HTTP/1.1.
X-Forwarded-For: <client>.
X-Forwarded-Proto: https.
X-Forwarded-Port: 443.
X-Forwarded-Host: <knox-server>.
X-Forwarded-Server: <knox-server>.
X-Forwarded-Context: /gateway/default.
User-Agent: curl/7.54.0.
Accept: */*.
Host: <myservice>:8090.
Connection: Keep-Alive.
Accept-Encoding: gzip,deflate.
.
#
T <myservice>:8090 -> <knox-server>:59118 [AP]
HTTP/1.1 200 OK.
Date: Sat, 14 Oct 2017 14:27:58 GMT.
X-Application-Context: myservice:prod:8090.
Content-Type: text/plain;charset=utf-8.
Content-Length: 4.
.
PONG
How should I configure Knox (0.12.0 from HDP 2.6.2) to make it pass Authorization header to the backend for websocket connection?
While writing this question I realised that there is a ticket KNOX-895 resolving the issue of passing cookies and headers to a backend service in Knox 0.14.0.
[EDIT]
I cloned knox git repo (commit 92b1505a), which includes KNOX-895 (2d236e78), run it locally with added websocket service to sandbox topology.
[tulinski]$ wscat -n --auth 'user:password' -c wss://localhost:8443/gateway/sandbox/echows
[tulinski]$ sudo ngrep -W byline host echo.websocket.org
#
T 192.168.0.16:59952 -> 174.129.224.73:80 [AP]
GET / HTTP/1.1.
Host: echo.websocket.org.
Upgrade: websocket.
Connection: Upgrade.
Sec-WebSocket-Key: Z4Qa9Dxwr6Qvq2QAicsT5Q==.
Sec-WebSocket-Version: 13.
Pragma: no-cache.
Cache-Control: no-cache.
Authorization: Basic dXNlcjpwYXNzd29yZA==.
.
##
T 174.129.224.73:80 -> 192.168.0.16:59952 [AP]
HTTP/1.1 101 Web Socket Protocol Handshake.
Connection: Upgrade.
Date: Mon, 16 Oct 2017 14:23:49 GMT.
Sec-WebSocket-Accept: meply+6cIyjbH+Vk2OsAqKJDWic=.
Server: Kaazing Gateway.
Upgrade: websocket.
.
Authorization header is passed to the backend service.
I successfully used Letsencrypt to generate certificates and I uploaded them to Heroku using:
this-site ********$ heroku addons:create ssl:endpoint
Creating ssl-graceful-41756... done, ($20.00/month)
Adding ssl-graceful-41756 to this-site... done
Next add your certificate with `heroku certs:add CERT KEY`.
Use `heroku addons:docs ssl` to view documentation.
this-site ********$ sudo heroku certs:add /etc/letsencrypt/live/www.this-site.com/fullchain.pem /etc/letsencrypt/live/www.this-site.com/privkey.pem
Resolving trust chain... done
Adding SSL Endpoint to this-site... done
this-site now served by qwasf-34234.herokussl.com
Certificate details:
Common Name(s): www.this-site.com
Expires At: 2016-09-02 19:15 UTC
Issuer: /C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
Starts At: 2016-06-04 19:15 UTC
Subject: /CN=www.this-site.com
SSL certificate is verified by a root authority.
However, when I visit qwasf-34234.herokussl.com, it is not working. It has a page that says: Heroku | No such app ; There is no app configured at that hostname.
Perhaps the app owner has renamed it, or you mistyped the URL.
I am copy and pasting the exact new host that heroku gave me. Going to https://qwasf-34234.herokussl.com yields the same page.
I verified the certificate with:
this-site ********$ heroku certs
Endpoint Common Name(s) Expires Trusted
-------------------------- --------------------- -------------------- -------
qwasf-34234.herokussl.com www.this-site.com 2016-09-02 19:15 UTC True
More checks:
this-site *******$ curl -kvI https://www.this-site.com
* Rebuilt URL to: https://www.michaelsutyak.com/
* Trying 23.21.142.230...
* Connected to www.this-site.com (23.21.142.230) port 443 (#0)
* TLS 1.2 connection using TLS_********************
* Server certificate: *.herokuapp.com
* Server certificate: DigiCert ******
* Server certificate: DigiCert *******
> HEAD / HTTP/1.1
> Host: www.this-site.com
> User-Agent: curl/7.43.0
> Accept: */*
>
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< Server: Cowboy
Server: Cowboy
< Connection: keep-alive
Connection: keep-alive
< Vary: Accept-Encoding
Vary: Accept-Encoding
< Content-Type: text/html; charset=utf-8
Content-Type: text/html; charset=utf-8
< Date: Sat, 04 Jun 2016 20:57:00 GMT
Date: Sat, 04 Jun 2016 20:57:00 GMT
< Via: 1.1 vegur
Via: 1.1 vegur
<
* Connection #0 to host www.this-site.com left intact
What is going on here and how can this work? I just want https for my site.
You cannot access the Heroku SSL endpoint directly. That endpoint represents the hostname where you need to point your domain to, as explained in the DNS and domain configuration of the Heroku article.
If you want to point a subdomain (e.g. www.this-site.com), then create a DNS record CNAME in your DNS hosting provider that points the www record to the Heroku SSL endpoint:
www CNAME qwasf-34234.herokussl.com
If you want to point the root domain (this-site.com), then you need to use a provide that supports the CNAME-like record for the root domain, as explained in this Heroku article as you can't use a CNAME for the root domain.
Make sure your domain is not still pointing to the herokuapp.com hostname.
You can test my assertion by sending a cURL request to the SSL endpoint, but passing the Host header (as the browser would do).
$ curl -i qwasf-34234.herokussl.com -H "Host: www.this-site.com"
You cannot visit the qwasf-34234.herokussl.com domain that Heroku gives you. Instead, you are supposed to change your DNS to point to that as a CNAME, instead of qwasf-34234.herokuapp.com.
I was writing a very simple Golang script and use this library golang-jenkins to connect with our internal HTTPS server. But I face the following x509 cert issue and wasn't sure what to do with the x509 cert problem. Our team has zero access to Jenkins and would like to know what else we can do to dig more about the issue.
$ go run jenkins.go
2014/07/28 22:00:29 [] Get https://jenkins.mydomain.com/api/json: x509: certificate signed by unknown authority (possibly because of "x509: cannot verify signature: algorithm unimplemented" while trying to verify candidate authority certificate "MyDomain Internal Root CA")
using curl:
$ curl -v "https://jenkins.mydomain.com/api/json"
* Adding handle: conn: 0x7f8469004000
* Adding handle: send: 0
* Adding handle: recv: 0
* Curl_addHandleToPipeline: length: 1
* - Conn 0 (0x7f8469004000) send_pipe: 1, recv_pipe: 0
* About to connect() to jenkins.mydomain.com port 443 (#0)
* Trying 10.38.8.70...
* Connected to jenkins.mydomain.com (10.38.8.70) port 443 (#0)
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
* Server certificate: jenkins.mydomain.com
* Server certificate: MyDomain Server CA - 2014
* Server certificate: MyDomain Internal Root CA
> GET /api/json HTTP/1.1
> User-Agent: curl/7.30.0
> Host: jenkins.mydomain.com
> Accept: */*
>
< HTTP/1.1 200 OK
* Server nginx is not blacklisted
< Server: nginx
< Date: Tue, 29 Jul 2014 05:03:45 GMT
< Content-Type: application/json;charset=UTF-8
< Transfer-Encoding: chunked
< Connection: keep-alive
< Set-Cookie: JSESSIONID.214ca1a4=1ry000odf815goiv7vl8tr627;Path=/;Secure
< Expires: Thu, 01 Jan 1970 00:00:00 GMT
< X-Jenkins: 1.554.3
< X-Jenkins-Session: c660ff91
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 is currently broken in Go, it will be supported in v1.4, the only workaround is to downgrade the TLS MaxVersion.
A quick look at golang-jenkins, it doesn't allow specifying the http.Client to use and just uses http.DefaultClient, the only ugly way to downgrade TLS's MaxVersion is to override http.DefaultClient.Transport.
You should be able to do something like this in func init() before you try to connect to anything:
cfg := &tls.Config{
MaxVersion: tls.VersionTLS11, // try tls.VersionTLS10 if this doesn't work
PreferServerCipherSuites: true,
}
http.DefaultClient.Transport = &http.Transport{
TLSClientConfig: cfg,
}
Keep in mind this will set the transport for anything that uses http.DefaultClient directly, like http.Get, however if you use your own instance, you will be fine.
Discussion about the bug: https://groups.google.com/forum/#!topic/golang-nuts/oK3EBAY2Uig