I'm looking for a solution to prevent a Jetty server to be taken down by a DDoS or similar. Currently the servlets will open a new thread for each incomming connections, so 1 mio incomming connections will open 1 mio threads and Jetty will explode.
What's the best way to avoid this thread? I thought about putting an Apache between client and server, since the webserver has the abilities to limit incomming connections from one ip to e.g. 5 connections/second.
What do you think about my idea?
Kind Regards,
Hendrik
Jetty ships with a Quality of Service filter that should do what you want.
See http://wiki.eclipse.org/Jetty/Feature/Quality_of_Service_Filter
DosFilter can be used to provide DDoS protection.
To quote the description from the wiki,
The Denial of Service (DoS) filter limits exposure to request
flooding, whether malicious, or as a result of a misconfigured client.
The DoS filter keeps track of the number of requests from a connection
per second. If the requests exceed the limit, Jetty rejects, delays,
or throttles the request, and sends a warning message.
To enable you have to include the below in the configuration in the webapp's web.xml or jetty-web.xml
<filter>
<filter-name>DoSFilter</filter-name>
<filter-class>org.eclipse.jetty.servlets.DoSFilter</filter-class>
<init-param>
<param-name>maxRequestsPerSec</param-name>
<param-value>30</param-value>
</init-param>
</filter>
Check the wiki for customization.
Idea with serving new connections with org.eclipse.jetty.servlets.QoSFilter is good but i rather use typical anti ddos configuration, based on iptables (like in this article: http://blog.bodhizazen.net/linux/prevent-dos-with-iptables/).
sudo iptables -A INPUT -p tcp --dport 80 -m state --state NEW -m limit --limit 50/minute --limit-burst 200 -j ACCEPT
sudo iptables -A INPUT -m state --state RELATED,ESTABLISHED -m limit --limit 50/second --limit-burst 50 -j ACCEPT
In this case ddos protection is separated from app, and is more productive because extra packages will drop before accessing jetty.
Related
I want to use Envoy as an L7 Load balancer implementation running alongside kubernetes, but I cannot give it admin access, as that is a security problem I have to worry about. Some research tells me that Envoy needs admin access to modify some IP tables, but do the other containers in the pod also need admin access? If so, why? And is that something for which a workaround exists?
Thank you
According to the instructions, you should create iptables rules with Envoy user UID (like in Istio).
sudo iptables -t nat -A PREROUTING -p tcp -j REDIRECT --to-port 5001
sudo iptables -t nat -A OUTPUT -p tcp -j REDIRECT --to-port 5001 -m owner '!' --uid-owner ${Envoy_User_UID}
In order to run this rule, a new user with ${Envoy_User_UID} needs to be created before launching Envoy. Envoy then needs to be run under this user id, so that packets from Envoy don't get redirected to themselves.
On the other hand, you can use Ambassador - it is an open source distribution of Envoy designed for Kubernetes.
How to deploy and configure Ambassador you can find in Envoy as an API Gateway in Kubernetes instruction.
You can find more details on Setup IP tables rules to redirect inbound/outbound traffic to Envoy sidecar page.
Also, Use Envoy as tcp/http proxy for all outbound traffic could be useful.
I'm trying to setup a haproxy gateway between server and client for full transparent proxy like below diagram. My main aim is to provide load balancing.
There is a simple application that listens port 25 in the server side. The client tries to connect port 25 on the gateway machine, and haproxy on the gateway chooses an avaliable server then redirects the connection to the server.
Network analysis of this approach produces tcp flow like diagram: The client resets the connection at the end since it doesn't send a syn packet to the server.
Is this haproxy usage true and my problem related configuation? Or should the client connect to the server directly (This doesn't make much sense to me but I'm not sure actually. If this is true then how haproxy will intervene the connection and make load balancing)?
EDIT:
I've started to think this problem is related to routing and NAT on the gateway. All of these three machines are in same subnetwork but I've added routes to the gateway for both client and server. Also rules on the gateway are:
iptables -t mangle -N DIVERT
iptables -t mangle -A DIVERT -j MARK --set-mark 0x01/0x01
iptables -t mangle -A DIVERT -j ACCEPT
iptables -t mangle -A PREROUTING -p tcp -m socket -j DIVERT
iptables -t mangle -A PREROUTING -p tcp --dport 25 -j TPROXY \
--tproxy-mark 0x1/0x1 --on-port 10025
ip route flush table 100
ip rule add fwmark 1 lookup 100
ip route add local 0.0.0.0/0 dev lo table 100
Now the question is what should I do in the gateway to change "syn-ack (src: S, dst: C)" to "syn-ack (src: GW, dst: C)"?
Here is the definition of my situation.
Here comes the transparent proxy mode: HAProxy can be configured to spoof the client IP address when establishing the TCP connection to the server. That way, the server thinks the connection comes from the client directly (of course, the server must answer back to HAProxy and not to the client, otherwise it can’t work: the client will get an acknowledge from the server IP while it has established the connection on HAProxy‘s IP).
And the answer is to set ip_nonlocal_bind system control.
I'm doing some loadbalancing testing with iptables by doing som portredirect on several ports on my server. i'm a bit curios how does the --random option work with the REDIRECT option?
is it using some kind of round robin algoritm? there doesn't seem to be any info about this in the man pages..
this is the iptable rule i'm using right now:
-A PREROUTING -d 10.10.10.1/32 -i eth0 -p udp -m udp --dport 5000 -j REDIRECT --to-ports 1194-1197 --random
also using this iptable rule will it take up alot of performance on the server? because I will have alot of traffic passing thorough the server.
The distribution is truly random, and it should not eat many resources by itself.
I am afraid about the utility of this spreading. Do you have e.g. 4 CPUs and 4 processes, each one listening to one port in 1194-1197 range?
I have a switch configured to mirror all traffic to an ethernet interface of a server. I can actually see the packets received with tshark, tcpdump, etc, but iptables doesn't seem to see this traffic. My ultimate goal is to ulog syn packets for connection accounting.
I tried to place rules in PREROUTING chain, unsuccessfully.
Can iptable capture packets not sent to the local machine? If no, is there a way to do this?
Which table do you use for monitoring?
What you want to do is to use the filter table (the default one) and the FORWARDING chain: it is specifically designed to capture packets which "traverse" the machine. For instance:
iptables -A FORWARDING -p tcp --dport 80 -j LOG
The INPUT chain will capture packets from the outside destined to the local machine, and the OUTPUT chain will capture packets originating from the machine and going outside.
One side note: packets transiting through loopback go through both INPUT and OUTPUT chains.
As to PREROUTING, it is a chain meant to modify packets, if necessary, before the routing decision -- this is why, for instance, port redirection is done in there. And this is why the filter table has no hook in it: it does not make sense.
iptables will only work with IP packets somehow directed at your machine. So what you are trying to achieve will not be doable with iptables. For it to work would require that you set up your accounting machine as a router for all IP traffic.
What’s wrong with tcpdump for this task?
tcpdump -G 3600 -w tcpsyn-%FT%T.pcap tcp and 'tcp[tcpflags] & (tcp-ack|tcp-syn) = tcp-syn'
If you want all TCP initiation attempts.
tcpdump -G 3600 -w tcpsynack-%FT%T.pcap tcp and 'tcp[tcpflags] & (tcp-ack|tcp-syn) = (tcp-ack|tcp-syn)'
If you want all TCP sessions actually established.
Here is the situation.
We have multiple server on our intranet 192.168.1.0/24
One of them is the default gateway for all of them and have two interfaces ($GATEWAY_INTERNAL_IP and $GATEWAY_EXTERNAL_IP).
We have also another server PUBLICHOST2 which has two IP as well $PUBLICHOST_EXTERNAL_IP and $PUBLICHOST_INTERNAL_IP.
We have a third server SERVER which have only one IP $PRIVIP and bind on port $PORT.
What we want is to be able to forward port $PORT on $PUBLICHOST_EXTERNAL_IP to host SERVER on $PRIVIP.
But when we do the port forwarding using iptables on PUBLICHOST2, SERVER receive the request but the response goes through the gateway and the connection is not successfull.
How can we properly do the setup so that the response can go back through PUBLICHOST2 ?
Thanks
You may need to set forwarding on for the interface. Try tne command.
sysctl -w net.ipv4.conf.eth0.forwarding=1
If you need additional help look for documentation on routeback or the Shorewall FAQ.
Well here what happens:
Client1 sends a request to PublicHost
The requests arrives and the iptables rules redirects the traffic (PAT) to the Server on the correct AppPort
Server sends back a reply to Client1 which will be routed by Gateway
Gateway is doing NAT and replaces the source IP with it's own
Client1 or Client1sGateway receives the IP packet with Gateway as the source but it expected PublicHost's IP in the source field of the IP packet.
Eventually Client1 resends the SYN/ACK (except if you're using a synproxy) to PublicHost and then drops the connection when whatever network related timer expires.
Now if you want to fix this, you should route all TCP traffic going OUT of Server and with a source port of AppPort to PublicHost.
If this doesn't work, PublicHost is not properly configured. Be sure to test the configuration with tcpdump.
I've been trying to do something similar. After running through a bunch of tutorials that never seemed to work until I Wiresharked the connection to discover that the destination address was still set to the external IP address, (exactly like you've described), I tried using the POSTROUTING chain to change the source IP address to that of the server:
iptables -t nat -A POSTROUTING -p <tcp/udp> --dport <destination_port> -j SNAT --to <$PUBLICHOST_INTERNAL_IP>
After I added that rule, the connection was forwarded into the private network and the response packets retraced the same path back to the client, rather than through the network gateway. I'm not positive what allowed the response packets back out through the firewall server, but I think it was because of the rule I already had on the INPUT chain to allow established connections:
iptables -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
The thing to be sure to keep in mind with this solution is: if you ever change the firewall server's internal IP address, then you will need to update the above POSTROUTING rule. (Needless to say, it's probably best if the firewall server has a statically assigned internal IP address).