I would like to run a small http proxy server on my machine which sends all requests to an upstream / cascading proxy server.
Twisted Web HTTP Proxy -> Squid -> Internet connection
Unfortunately I cannot figure out how to send all ProxyClient requests upstream / cascade them to Squid. Could you give me a hint?
Do I have to use reactor.connectTCP and HttpClientFactory for this and somehow direct the Squid response through?
I have used the following Twisted code sample to get the twisted http proxy up:
from twisted.web import proxy, http
from twisted.internet import reactor
from twisted.python import log
import sys
log.startLogging(sys.stdout)
class ProxyFactory(http.HTTPFactory):
protocol = proxy.Proxy
reactor.listenTCP(8080, ProxyFactory())
reactor.run()
Thanks,
Mathias
I think you're over complicating things. To me, it sounds like what you want to do is round-robin load balance http proxy requests between two squid services.
If it was me, I wouldn't write a byte of code and use something off the shelf.
I'd use HA-Proxy with a configuration something like this:
global
daemon
maxconn 256
defaults
mode http
timeout connect 5000ms
timeout client 50000ms
timeout server 50000ms
frontend loadbal
bind *:3128
default_backend squids
option http_proxy
backend squids
option http_proxy
server squid1 192.168.1.2:3128
server squid2 192.168.1.3:3128
balance roundrobin
You might need to do some playing around, some webservices may require certain types of stickiness, but this application is highly configurable in this regard. It looks like HA-Proxy v 1.4 supports keep alive connections, which may simplify some of the corner cases I can imagine.
Oh, also, if you're trying to get the most out of your two connections, I'd suggest that you configure ICP between your two squid boxes.
This sounds like something twisted.web.proxy.ReverseProxyResource might be good for.
Related
The Valhalla maps server docs assume that the server is always running on "http://[hostname]:8002"
(see https://github.com/valhalla/valhalla)
How can the server be configured to listen via SSL/https instead?
Is there any detailed documentation on how to do this?
Thnx.
To answer my own question:
After much reading & research I came to the conclusion that a practical way to achieve this is simply to hide the Valhalla port (port 8002) behind my Linux firewall and to expose port 443 (SSL) instead and have Nginx running on that port. Nginx then port-forwards the Valhalla request obj to the internal port 8002 and proxies the response back to the caller for the return journey on the encrypted channel.
Setting up Nginx to achieve this is fairly straightforward and the procedure is documented on many websites.
I'm trying to connect to the rabbitmq which is hidden behind nginx proxy. It's declared as:
location ^~ /rabbitmq/ {
proxy_pass http://127.0.0.1:5672/;
}
The problem is that as I found AMPQ only specifies host but it doesn't know anything about urls.
Can I connect rabbit client to www.myserver.com/rabbitmq somehow? I'm using EasyNetQ to connect, but it looks like a protocol limitation, and implementation doesn't matter.
If it's not possible at all maybe there are some workarounds?
For AMQP, If using Nginx probably doing a TCP load balancing could help: https://docs.nginx.com/nginx/admin-guide/load-balancer/tcp-udp-load-balancer/, otherwise if you could use HAProxy you could try something like this:
frontend rabbitmq
mode tcp
bind *:5672
use_backend bunny
backend bunny
mode tcp
server bunny 127.0.0.1:5672 check
If you want to publish message over HTTP probably you would like to expose the rabbitmq API:
http://localhost:15672/api/index.html
Notice the port 15672, from the docs:
Note that the UI and HTTP API port — typically 15672 — does not support AMQP 0-9-1, AMQP 1.0, STOMP or MQTT connections. Separate ports should be used by those clients.
I want to have one server with HAproxy and a standalone mod_security installed which routes every packets to mod_security first and check by its rules.
Then if there wasn't anything suspicious in packets (SQL Injection, DOS Attacks, ...) pass them back from mod_security to haproxy and haproxy routes them to multiple servers with different webservers.
Therefore I don't need to install and config mod_security on all my webservers.
This is technically possible, possibly with running 2 instances of HAProxy. However, you will need a webserver to run underneath ModSec, typically Apache or nginx, and this kind of negates the advantage of not having to install ModSec on all your webservers.
The standard setup is: haproxy -> reverse-proxies with ModSec -> application-servers
Just to answer this old, but still valid, question:
The solution should be to use HAProxies Stream Processing Offload Engine (SPOE) through the Stream Processing Offload Protocol (SPOP) to talk a Stream Processing Offload Agent (SPOA) which is a standalone modsecurity daemon.
HAProxy example config from their github repo
frontend my-front
...
filter spoe engine modsecurity config spoe-modsecurity.conf
...
enter code here
backend spoe-modsecurity
mode tcp
balance roundrobin
timeout connect 5s
timeout server 3m
server modsec1 127.0.0.1:12345
# Block potential malicious requests with returncode < 0
http-request deny if { var(txn.modsec.code) -m int gt 0 }
There's also a Github project where the daemon has been made available as Docker container
Offical HAProxy blog post
I found some inconvenience in haproxy 1.5 when i try to configure SSL SNI.
There is a fragment of haproxy configuration: pastebin
I would like to pass client IP to backend. No matter how I configure reqadd / set-header X-Forwarded-For / Real-IP I always got a haproxy IP address in X-Forwarded-For.
Someone try to pass real IP with SSL SNI on HAProxy
? :/
From this configuration, you seem to be doing SNI-sniffing, yet all of the backends are looping back to HAProxy itself... which is not a case where SNI-sniffing is required. Perhaps I'm overlooking something else that would require this.
It should be apparent why you are getting the proxy's IP in X-Forwarded-For -- HAProxy is talking to itself. The first pass through the proxy is the client connection, as far as the proxy can determine on the second pass, because only the second pass is speaking HTTP. It only sees that an incoming TCP connection has arrived... from itself.
The solution is for the first-pass backend to pass the original client information using the Proxy Protocol and the second-pass frontend to decode it.
Add accept-proxy to the bind lines for the second-pass frontends, and add send-proxy to the server lines on the first-pass backends. This way, on the connection where HAProxy is talking to itself, the first-pass backend will send the Proxy protocol preamble and the second-pass frontend will decode the incoming value and place it in X-Forwarded-For.
Le'ts say you open a tcp socket on port 80 to handle http request, and a ssl socket on port 443 to deal with https...how can some proxy provide access to both of them on the same port??
I found only this link but it wasn't very useful. Can you provide me an erlang example or suggest me some resources from which i can learn more on the topic?
Thanks in advance
how can some proxy provide access to both of them on the same port??
By implementing the HTTP CONNECT method, the (non-transparent) proxy may switch to providing a TCP tunnel over which a browser may, for example, access an HTTPS resource.
A rather sparse specification:
https://www.rfc-editor.org/rfc/rfc2616#section-9.9
As outlined in the link you provide, you will need to write your own custom server that sniffs the request and then redirects to the correct protocol accordingly.
As http://www.faqs.org/rfcs/rfc2818.html indicates, an HTTP session will start with an Initial Request Line (e.g. GET /), whereas a TLS session will start with a ClientHello (more on the TLS session on wikipedia)
There are lots of resources online about writing servers in Erlang, e.g. How to write a simple webserver in Erlang?
Incidentally your terminology is incorrect: http, https SSL and TLS are protocols, and all operate (over the web) using TCP sockets.