I would like a mod-security rule that takes a list of IP addresses from a text data file and if the Client IP does not match one of these, then rate limit requests to 200 requests-per-minute.
Don't use ModSecurity for this. It's not great at handling persistent variables between requests - which is needed for any type of rate limiting. The functionality is there, but because of of the disk based SDBM disk based storage it uses to implement this, this doesn't work under any real load. See this discussion on ModeSecurity mailing thread as one of many examples threads on this subject.
To me this will not really be an option in ModSecurity until some non-disk based storage is used, so best to keep an eye on this bug to see when that is implemented.
Instead look at fail2ban or some other firewall protection for this.
Related
Right now, I am working on an ASP.NET Core Web API that calls an external web service and uses the returned data in its own response. This is working fine.
However, I discovered that the external service is not as scalable as I would like to. Therefore, as discussed with the company providing this external service, the number of outgoing requests needs to be limited to one per second. I als use caching to reduce the number of outgoing requests but this has been shown to be not effective enough because (as logically) it only works when a lot of requests are the same so cache data can be reused.
I have been doing some investigation on rate limiting but the documented examples and types are far more complicated than what I need. This is not about partitions, tokens or concurrency. What I want is far more simple. Just 1 outgoing request per second and that´s all.
I am not that experienced when it comes to rate limiting. I just started reading the referred documentation and found out that there is a nice and professional package for it. But the examples are more complicated than what I need for the reasons explained. It is not about tokens or concurrency or so. It is the number of outgoing requests per second that needs to be limited.
Possibly, there is a way using the package System.Threading.RateLimiting in such a way that this is possible by applying the RateLimiter class correctly. Otherwise, I may need to write my own DelegateHandler implementation to do this. But there must be a straightforward way which people with experience in rate limiting can explain me.
So how to limit the number of outgoing web request per second?
In addition, what I want to prevent is a 429 or so in case of to many request. In such a situation, the process should just take more waiting time in order to complete so the number of outgoing requests is limited.
I implemented the Express-rate-limit npm module on my code (nodejs)
I saw the DDOS Module.
Anyone who have good expertise on Nodejs please suggest me that wheher I have to use DDOS module or not.
I installed the module but this will block the request. I read about express rate-limit also, this module is also working same as DDOS.
Someone suggest me that use DDOS. I told that I already used Express-Rate-Limit but he said that Use this also.
I am confused now. Please give me the Proper input regarding this. Any help is really appreciate.
it's fine as basic shield from ddos, or handling external requests for your api methods, that can go-out-of limit.
But if you want to prevent real ddos attacks, your should check debouncing and event throttling. Also think about per-machine custom firewall configurations;)
Dig a bit more into docs of this module ;)
burst Burst is the number or amount of allowable burst requests before
the client starts being penalized. When the client is penalized, the
expiration is increased by twice the previous expiration.
bursts = base request counter for 1 unit of time, defined by default as 1 second, or a custom set up
limit
limit is the number of maximum counts allowed (do not confuse that
with maxcount). count increments with each request. If the count
exceeds the limit, then the request is denied. Recommended limit is to
use a multiple of the number of bursts.
requests received => check for the limit. If limit achieved, requester gets a penalty.
When you see a lot of requests(multiple bursts detected).
That's real detection for excide of request limit.
So, 5 bursts set, 20 as limit, when burst detected as 5, it will flag 20 request counter like a fully recognized limitation
maxexpiry
maxexpiry is the seconds of maximum amount of expiration time. In
order for the user to use whatever service you are providing again,
they have to wait through the expiration time.
And that's it. Just dive into testing this stuff;)
I need to get an understanding of ISO-8583 message platform,lets say i want to perform a authorization of a card transaction,so in real time at a particular instance lets say i got 100000 requests from network(VISA/MASTERCARD) all for authorization,how do i define priority of there request and the response,can the connection pool handle it(in my case its HIKARI),how is it done banks/financial institutions for authorizing a request.Please provide me some insights on how to manage all these requests.Should i go for a MQ?
Tech used are:-spring boot,hibernate,spring-tcp-starter
Your question doesn't seem to be very well researched as there are a ton of switch platforms out there that due this today and many of their technology guides can be found on the web including for major vendors like ACI, FIS, AJB,.. etc if you look yard enough.
I have worked with several iso-interface specifications, commercial switches, and home grown platforms and it is actually pretty consistent in how they do the core realtime processing.
This information on prioritization is generally in each ISO-8583 message processing specification and is made explicitly clear in almost every specification I've ever read written by someone who is familar with ISO-8533 and not just making up their own variant or copying someone elses.
That said.. in general at a high level authorizations / financials (0100, 0200) requests always have high priority than force posts (0x20) messages.
Administrative messages in the 05xx and 06xx and 08xx sometimes also get bumped up above other advices.. but these are still advices and almost always auths/financials are always processed first as they A) Impact the customer B) have much tighter timers than any advice by usually more than double or more.
Most switches I have seen do it entirely in memory without going to MQ and or some other disk for core authorization process to manage these.. but not to say there is not some sort of home grown middle ware sometimes involved.. but non-realtime processes regularly use a MQ process to queue or disk queuing these up into processes not in-line of the approval for this Store-and-forward (SAF) processing.. but many of these still use memory only processing for the front of their queue.
It is important to also differentiate between 100000 requests and 100000 transactions.. the various exchanges both internal and external make a big difference in the number of actual requests/responses in flight at even given time.. a basic transaction can be accomplished in like two messages.. but some of the more complex ones can easily exceed 20 messages just for a pre-authorization or a completion component.
If you are dealing with largely batch transaction bursts.. I can see the challenge of queuing but almost every application I have seen has a max in flight for advices and requests separate of each other.. and sometimes even with different timers.. and the apps pumping the transactions almost always wait for the response back before sending more.. and this tends to work fine for just about everyone.. including big posting batches from retailers and card networks. So if your app doesn't have them.. you probably need to add them.
In fact your 100000 requests should be sorted by (Terminal ID and/or Merchant ID) + (timestamp/local timestamp) + (STAN and/or RRN).
Duplicated transaction requests expected to be rejected.
If you simulating multiple requests from single terminal (or host) with same test card details the increasing of STAN/RRN would be a case.
Please refer to previous answers about STAN and RRN ISO 8583 fields.
In ISO message, what's the use of stan and rrn ?
I have file download site. What I look for is limiting bandwidth per IP (!). Limit should be set dynamically by HTTP header from backend.
My current implementation uses X-Accel-Limit-Rate (I can change that header, it's not hard-coded anywhere), but it does limit only current connection/request.
Is my idea doable in G-Wan?
Yes, this can be done.
Write a G-WAN handler to extract the X-Accel-Limit-Rate HTTP header. Then enforce this policy by using the throttle_reply() G-WAN API call documented here.
An example available called throttle.c might help you further.
The throttle_reply() G-WAN function lets you apply throttling on a global basis or per connection, so you will just apply the relevant throttling values for either IP addresses or authenticated users, depending on your needs.
throttle_reply() can change the download speed dynamically during the lifespan of each connection so you can slow-down old connections and create new ones with an adaptive download rate.
Of course, this can be enforced on a per client IP address (or cookie, or even ISP/Datacenter AS record) to deal with huge workloads.
We've an existing system which connects to the the back end via http (apache/ssl) and polls the server for new messages, needless to say we have scalability issues.
I'm researching on removing this polling and have come across BOSH/XMPP but I'm not sure how we should take the BOSH technique (using long lived http connection).
I've seen there are few libraries available but the entire thing seems bloaty since we do not need buddy lists etc and simply want to notify the clients of available messages.
The client is written in C/C++ and works across most OS so that is an important factor. The server is in Java.
does bosh result in huge number of httpd processes? since it has to keep all the clients connected, what would be the limit on that. we are also planning to move to 64 bit JVM/apache what would be the max limit of clients in that case.
any hints?
I would note that BOSH is separate from XMPP, so there's no "buddy lists" involved. XMPP-over-BOSH is what you're thinking of there.
Take a look at collecta.com and associated blog posts (probably by Jack Moffitt) about how they use BOSH (and also XMPP) to deliver real-time information to large numbers of users.
As for the scaling issues with Apache, I don't know — presumably each connection is using few resources, so you can increase the number of connections per Apache process. But you could also check out some of the connection manager technologies (like punjab) mentioned on the BOSH page above.