i am writing my own ReverseProxy in Go.The ReverseProxy should connect my go-webserver and my apache2 webserver. But when I run my reverseproxy on another IP-Adress then my Apache2 webserver I got following error in my apache-logfile, when the reverseproxy sends the request to apache.
"Hosname xxxx provided via sni and hostname xxxx2 provided via http are different"
My Reverse Proxy and apache-webserver running on https.
Here some code:
func (p *Proxy) directorApache(req *http.Request) {
mainServer := fmt.Sprintf("%s:%d", Config.HostMain, Config.PortMain)
req.URL.Scheme = "https"
req.URL.Host = mainServer
}
func (p *Proxy) directorGo(req *http.Request) {
goServer := fmt.Sprintf("%s:%d", Config.GoHost, Config.GoPort)
req.URL.Scheme = "http"
req.URL.Host = goServer
}
func (p *Proxy) ServeHTTP(rw http.ResponseWriter, req *http.Request) {
fmt.Println(req.URL.Path)
if p.isGoRequest(req) {
fmt.Println("GO")
p.goProxy.ServeHTTP(rw, req)
return
}
p.httpProxy.ServeHTTP(rw, req)
}
func main() {
var configPath = flag.String("conf", "./configReverse.json", "Path to the Json config file.")
flag.Parse()
proxy := New(*configPath)
cert, err := tls.LoadX509KeyPair(Config.PathCert, Config.PathPrivateKey)
if err != nil {
log.Fatalf("server: loadkeys: %s", err)
}
config := tls.Config{InsecureSkipVerify: true, Certificates: []tls.Certificate{cert}}
listener, err := net.Listen("tcp",
net.JoinHostPort(proxy.Host, strconv.Itoa(proxy.Port)))
if err != nil {
log.Fatalf("server: listen: %s", err)
}
log.Printf("server: listening on %s")
proxy.listener = tls.NewListener(listener, &config)
serverHTTPS := &http.Server{
Handler: proxy.mux,
TLSConfig: &config,
}
if err := serverHTTPS.Serve(proxy.listener); err != nil {
log.Fatal("SERVER ERROR:", err)
}
}
Perhaps someone has a idea about that issue.
Short example
Say you're starting an HTTP request to https://your-proxy.local. Your request handler takes the http.Request struct and rewrites its URL field to https://your-apache-backend.local.
What you have not considered, is that the original HTTP request also contained a Host header (Host: your-proxy.local). When passing that same request to http://your-apache-backend.local, the Host header in that request still says Host: your-proxy.local. And that's what Apache is complaining about.
Explanation
As you're using TLS with Server Name Indication (SNI), the request hostname will not only be used for DNS resolution, but also to select the SSL certificate that should be used to establish the TLS connection. The HTTP 1.1 Host header on the other hand is used to distinguish several virtual hosts by Apache. Both names must match. This issue is also mentioned in the Apache HTTPD wiki:
SNI/Request hostname mismatch, or SNI provides hostname and request doesn't.
This is a browser bug. Apache will reject the request with a 400-type error.
Solution
Also rewrite the Host header. If you want to preserve the original Host header, you can store it in an X-Forwarded-Host header (that's a non-standard header, but it's widely used in reverse proxies):
func (p *Proxy) directorApache(req *http.Request) {
mainServer := fmt.Sprintf("%s:%d", Config.HostMain, Config.PortMain)
req.URL.Scheme = "https"
req.URL.Host = mainServer
req.Header.Set("X-Forwarded-Host", req.Header().Get("Host"))
req.Host = mainServer
}
Related
I have a simple proxy that listens to HTTP and HTTPS connections on the same port.
However I'm seeing weird TLS handshake errors that seem to be caused by the Heroku router:
heroku[router]: sock=backend at=error code=H18 desc="Server Request Interrupted" method=GET path="/favicon.ico" host=banana.camp request_id=f56144c8-e480-476a-90b8-429b490f1ff5 fwd="24.67.185.77" dyno=web.1 connect=0ms service=1ms status=503 bytes=7 protocol=https
http: TLS handshake error from 10.81.159.108:19578: tls: first record does not look like a TLS handshake
http: TLS handshake error from 172.17.117.25:36924: EOF
Is there any way to fix or ignore them? Any clues are greatly appreciated.
Thanks!
func InitServer() error {
m := autocert.Manager{
Cache: autocert.DirCache(*StorageDir),
Prompt: autocert.AcceptTOS,
HostPolicy: func(ctx context.Context, host string) error {
return nil
},
}
errchan := make(chan error)
s := &http.Server{
Addr: ":" + os.Getenv("PORT"),
TLSConfig: &tls.Config{GetCertificate: m.GetCertificate},
Handler: ServeHTTP(),
}
go (func() {
errchan <- http.ListenAndServe(":"+os.Getenv("PORT"), m.HTTPHandler(nil))
})()
errchan <- s.ListenAndServeTLS("", "")
return <-errchan
}
func ServeHTTP() http.Handler {
return http.HandlerFunc(func(res http.ResponseWriter, req *http.Request) {
if upstreams, ok := Hosts[req.Host]; ok {
forwarder, _ := forward.New(forward.PassHostHeader(true))
loadbalancer, _ := roundrobin.New(forwarder)
for _, upstream := range upstreams {
if url, err := url.Parse(upstream); err == nil {
loadbalancer.UpsertServer(url)
} else {
log.Fatal("InitHostsList: ", err)
}
}
loadbalancer.ServeHTTP(res, req)
return
}
http.Error(res, "The request service couldn't be found here", http.StatusNotImplemented)
})
}
TLS termination is handled by the Heroku router and can't be done at the app level ¯_(ツ)_/¯
You can't listen to both http and https on the same port, and those log-errors are cases when you have http clients try to connect to the app.
By default, the golang http server only supports listening to http or https traffic on a given port (not both). To listen to https traffic, you need to use http.ListenAndServeTLS.
If you are directing both http and https traffic at at point configured for only one, you will end up seeing errors.
There are third-party solution (like cmux) that support hosting both secure and insecure traffic on the same port. Not as easy as separating the ports, but if it's something you want to do, they have an example configuration in the README.
I have a http2 server but by default it responds http1 requests.
I want to stop my server responding to http1 requests?
Most browsers might use alpn or npn. is there a possibility to advertise only http2 ? or a custom list of application protocols ?
In Caddy, if you're comfortable modifying the source code, you can make the following changes in caddyhttp/httpserver/server.go.
Change the line that says:
var defaultALPN = []string{"h2", "http/1.1"}
…so that it says:
var defaultALPN = []string{"h2"}
This will prevent it from advertising HTTP/1.1 via ALPN.
Then add this code to the beginning of the ServeHTTP method:
func (s *Server) ServeHTTP(w http.ResponseWriter, r *http.Request) {
if !r.ProtoAtLeast(2, 0) {
if hj, ok := w.(http.Hijacker); ok {
conn, _, err := hj.Hijack()
if err == nil {
conn.Close()
}
}
return
}
…
}
This will immediately close the connection without sending headers if any protocol earlier than HTTP/2 is used.
You could use something like below rule to block all http1 requests. But it is not advisable to do this.
if ($server_protocol ~* "HTTP/1*") {
return 444;
}
I am trying to make a GET request against an HTTPS URL using proxy with username/password authorization (auth is required by the proxy not the website).
Here's what I do:
package main
import (
"crypto/tls"
"encoding/base64"
"fmt"
"net/http"
"net/http/httputil"
"net/url"
)
func main() {
ua := "Mozilla/5.0 (Windows NT 6.1"
basic := "Basic " + base64.StdEncoding.EncodeToString([]byte("username:mypass"))
req, err := http.NewRequest("GET", "https://api.ipify.org/", nil)
proxyUrl, _ := url.Parse("http://myproxy.com:9999")
fmt.Println(basic) // Basic dXNlcm5hbWU6bXlwYXNz
req.Header.Add("Proxy-Authorization", basic)
req.Header.Add("User-Agent", ua)
bb, _ := httputil.DumpRequest(req, false)
fmt.Println(string(bb))
/*
Get / HTTP/1.1
Host: api.ipify.org
Proxy-Authorization: Basic dXNlcm5hbWU6bXlwYXNz
User-Agent: Mozilla/5.0 (Windows NT 6.1
*/
client := &http.Client{
Transport: &http.Transport{
TLSClientConfig: &tls.Config{InsecureSkipVerify: true},
Proxy: http.ProxyURL(proxyUrl),
},
}
resp, err := client.Do(req)
fmt.Println(err) // Proxy Authentication Required
fmt.Println(resp) // <nil>
}
The catch is that when I try to do a request to an HTTP (not HTTPS) site it goes fine, but if I make HTTPS request it fails (see above the message).
I tested the proxy with my browser (FireFox) and everything goes well, I looked for the headers through firebug and added everything relevant to the request (above). I've double-triple-checked the basic value and everything else but without any luck.
So does some one have any idea why this happens or at least how do I research the problem?
The last thing to add is that I can use a public HTTP proxy (that doesn't require any auth) in this case and problems seem to start when auth enters in this process (the error also suggests that).
P.S. Unfortunately I cannot share the proxy IP, port and username cause it is against their policy.
package main
import (
"crypto/tls"
"fmt"
"net/url"
"net/http"
)
func main() {
req, err := http.NewRequest("GET", "https://api.ipify.org/", nil)
proxyUrl, _ := url.Parse("http://username:password#127.0.0.1:9999")
client := &http.Client{
Transport: &http.Transport{
TLSClientConfig: &tls.Config{InsecureSkipVerify: true},
Proxy: http.ProxyURL(proxyUrl),
},
}
_, err = client.Do(req)
fmt.Println(err)
}
i am trying to make one get in https://www.google.com using proxy with authentication, i already passing header parameter
Proxy-Authorization
but proxy server return
Proxy Authentication Required
code:
package main
import (
"crypto/tls"
"encoding/base64"
"fmt"
"io/ioutil"
"net/http"
"net/url"
"os"
)
func main() {
req, _ := http.NewRequest("GET", "https://www.google.com.br", nil)
req.Header.Set("Host", "www.google.com.br")
proxyURL := url.URL{
Host: "IP-HERE:PORT-HEERE"}
transport := &http.Transport{
Proxy: http.ProxyURL(&proxyURL),
TLSClientConfig: &tls.Config{},
}
client := &http.Client{Transport: transport}
req.RequestURI = ""
auth := fmt.Sprintf("USER:PASSWORD")
basic := "Basic " + base64.StdEncoding.EncodeToString([]byte(auth))
req.Header.Add("Proxy-Authorization", basic)
resp, err := client.Do(req)
if err != nil {
fmt.Printf("erro: %s", err)
}
fmt.Printf("code: %s", resp.StatusCode)
htmlData, err := ioutil.ReadAll(resp.Body)
if err != nil {
fmt.Println(err)
os.Exit(1)
}
fmt.Println(os.Stdout, string(htmlData))
}
I have to pass another parameter?
When i perform one get in http://www.google.com.br, without https.. proxy authentication with success. why?
I found the solution. Golang don't pass header parameters in method CONNECT, then parameter Proxy-Authorization aren't sent to proxy server.
to resolve this, field ProxyConnectHeader was added to struct Transport. but this change until released, this change are only in master.
to test this new field using golang from master, i made on project in github and worked.
link to golang path
link to project in github that test this new field
I'm trying to write tests for a package that makes requests to a web service. I'm running into issues probably due to my lack of understanding of TLS.
Currently my test looks something like this:
func TestSimple() {
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(200)
fmt.Fprintf(w, `{ "fake" : "json data here" }`)
}))
transport := &http.Transport{
Proxy: func(req *http.Request) (*url.URL, error) {
return url.Parse(server.URL)
},
}
// Client is the type in my package that makes requests
client := Client{
c: http.Client{Transport: transport},
}
client.DoRequest() // ...
}
My package has a package variable (I'd like for it to be a constant..) for the base address of the web service to query. It is an https URL. The test server I created above is plain HTTP, no TLS.
By default, my test fails with the error "tls: first record does not look like a TLS handshake."
To get this to work, my tests change the package variable to a plain http URL instead of https before making the query.
Is there any way around this? Can I make the package variable a constant (https), and either set up a http.Transport that "downgrades" to unencrypted HTTP, or use httptest.NewTLSServer() instead?
(When I try to use NewTLSServer() I get "http: TLS handshake error from 127.0.0.1:45678: tls: oversized record received with length 20037")
Most of the behavior in net/http can be mocked, extended, or altered. Although http.Client is a concrete type that implements HTTP client semantics, all of its fields are exported and may be customized.
The Client.Transport field, in particular, may be replaced to make the Client do anything from using custom protocols (such as ftp:// or file://) to connecting directly to local handlers (without generating HTTP protocol bytes or sending anything over the network).
The client functions, such as http.Get, all utilize the exported http.DefaultClient package variable (which you may modify), so code that utilizes these convenience functions does not, for example, have to be changed to call methods on a custom Client variable. Note that while it would be unreasonable to modify global behavior in a publicly-available library, it's very useful to do so in applications and tests (including library tests).
http://play.golang.org/p/afljO086iB contains a custom http.RoundTripper that rewrites the request URL so that it'll be routed to a locally hosted httptest.Server, and another example that directly passes the request to an http.Handler, along with a custom http.ResponseWriter implementation, in order to create an http.Response. The second approach isn't as diligent as the first (it doesn't fill out as many fields in the Response value) but is more efficient, and should be compatible enough to work with most handlers and client callers.
The above-linked code is included below as well:
package main
import (
"fmt"
"io"
"log"
"net/http"
"net/http/httptest"
"net/url"
"os"
"path"
"strings"
)
func Handler(w http.ResponseWriter, r *http.Request) {
fmt.Fprintf(w, "hello %s\n", path.Base(r.URL.Path))
}
func main() {
s := httptest.NewServer(http.HandlerFunc(Handler))
u, err := url.Parse(s.URL)
if err != nil {
log.Fatalln("failed to parse httptest.Server URL:", err)
}
http.DefaultClient.Transport = RewriteTransport{URL: u}
resp, err := http.Get("https://google.com/path-one")
if err != nil {
log.Fatalln("failed to send first request:", err)
}
fmt.Println("[First Response]")
resp.Write(os.Stdout)
fmt.Print("\n", strings.Repeat("-", 80), "\n\n")
http.DefaultClient.Transport = HandlerTransport{http.HandlerFunc(Handler)}
resp, err = http.Get("https://google.com/path-two")
if err != nil {
log.Fatalln("failed to send second request:", err)
}
fmt.Println("[Second Response]")
resp.Write(os.Stdout)
}
// RewriteTransport is an http.RoundTripper that rewrites requests
// using the provided URL's Scheme and Host, and its Path as a prefix.
// The Opaque field is untouched.
// If Transport is nil, http.DefaultTransport is used
type RewriteTransport struct {
Transport http.RoundTripper
URL *url.URL
}
func (t RewriteTransport) RoundTrip(req *http.Request) (*http.Response, error) {
// note that url.URL.ResolveReference doesn't work here
// since t.u is an absolute url
req.URL.Scheme = t.URL.Scheme
req.URL.Host = t.URL.Host
req.URL.Path = path.Join(t.URL.Path, req.URL.Path)
rt := t.Transport
if rt == nil {
rt = http.DefaultTransport
}
return rt.RoundTrip(req)
}
type HandlerTransport struct{ h http.Handler }
func (t HandlerTransport) RoundTrip(req *http.Request) (*http.Response, error) {
r, w := io.Pipe()
resp := &http.Response{
Proto: "HTTP/1.1",
ProtoMajor: 1,
ProtoMinor: 1,
Header: make(http.Header),
Body: r,
Request: req,
}
ready := make(chan struct{})
prw := &pipeResponseWriter{r, w, resp, ready}
go func() {
defer w.Close()
t.h.ServeHTTP(prw, req)
}()
<-ready
return resp, nil
}
type pipeResponseWriter struct {
r *io.PipeReader
w *io.PipeWriter
resp *http.Response
ready chan<- struct{}
}
func (w *pipeResponseWriter) Header() http.Header {
return w.resp.Header
}
func (w *pipeResponseWriter) Write(p []byte) (int, error) {
if w.ready != nil {
w.WriteHeader(http.StatusOK)
}
return w.w.Write(p)
}
func (w *pipeResponseWriter) WriteHeader(status int) {
if w.ready == nil {
// already called
return
}
w.resp.StatusCode = status
w.resp.Status = fmt.Sprintf("%d %s", status, http.StatusText(status))
close(w.ready)
w.ready = nil
}
The reason you're getting the error http: TLS handshake error from 127.0.0.1:45678: tls: oversized record received with length 20037 is because https requires a domain name (not an IP Address). Domain names are SSL certificates are assigned to.
Start the httptest server in TLS mode with your own certs
cert, err := tls.LoadX509KeyPair("cert.pem", "key.pem")
if err != nil {
log.Panic("bad server certs: ", err)
}
certs := []tls.Certificate{cert}
server = httptest.NewUnstartedServer(router)
server.TLS = &tls.Config{Certificates: certs}
server.StartTLS()
serverPort = ":" + strings.Split(server.URL, ":")[2] // it's always https://127.0.0.1:<port>
server.URL = "https://sub.domain.com" + serverPort
To provide a valid SSL certificate for a connection are the options of:
Not supplying a cert and key
Supplying a self-signed cert and key
Supplying a real valid cert and key
No Cert
If you don't supply your own cert, then an example.com cert is loaded as default.
Self-Signed Cert
To create a testing cert can use the included self-signed cert generator at $GOROOT/src/crypto/tls/generate_cert.go --host "*.domain.name"
You'll get x509: certificate signed by unknown authority warnings because it's self-signed so you'll need to have your client skip those warnings, by adding the following to your http.Transport field:
TLSClientConfig: &tls.Config{InsecureSkipVerify: true}
Valid Real Cert
Finally, if you're going to use a real cert, then save the valid cert and key where they can be loaded.
The key here is to use server.URL = https://sub.domain.com to supply your own domain.
From Go 1.9+ you can use func (s *Server) Client() *http.Client in the httptest package:
Client returns an HTTP client configured for making requests to the server. It is configured to trust the server's TLS test certificate and will close its idle connections on Server.Close.
Example from the package:
package main
import (
"fmt"
"io"
"log"
"net/http"
"net/http/httptest"
)
func main() {
ts := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
fmt.Fprintln(w, "Hello, client")
}))
defer ts.Close()
client := ts.Client()
res, err := client.Get(ts.URL)
if err != nil {
log.Fatal(err)
}
greeting, err := io.ReadAll(res.Body)
res.Body.Close()
if err != nil {
log.Fatal(err)
}
fmt.Printf("%s", greeting)
}