We have a scenario where a bunch of producers come online at the exact same time. These use librdkafka. We see that the flood of requests make Kafka open a bunch of file descriptors and eventually run of them and crash. I tried setting the request percentage quota but it looks like it is not applicable to auth requests. I also tried adjusting num.network.threads and num.io.threads but they don't seem to help after a certain limit. Any ideas on how I can throttle the auth requests? I am running v0.11.
Related
When Testcafe runs against our local site, every request it makes during the test steps are prepended with something like http://192.168.1.182:59304/http://localhost:3000 (port number varies per run).
For the most part this works, but our web application makes calls to certain APIs during a user journey, and within TestCafe they might look like: http://192.168.1.182:59304/http://www.example.com/api/v2/customers/1 which come back with a 401 and response body of 'unauthorized'. Some API calls are fine, however.
I guess my question is:
Are there any way to get around this from my side, such as rewrite certain requests, or do I need to contact the API provider - and if so, what would they be potentially looking to do to allow these requests to go ahead?
You have faced this issue: https://github.com/DevExpress/testcafe-hammerhead/issues/2344. It was fixed. Try to run your tests with the latest TestCafe version (1.8.8-alpha.3).
We are running an SPA that communicates with an API. Both are exposed to the public via Cloudfront.
We now have the issue that the requests we see in the backend are masked by Cloudfront. Meaning:
The Remote Address we see is the address of the AWS Cloud
The User Agent Header field is set to "Amazon Cloudfront" and not the browser of the user
So Cloudfront somehow intercepts the request in a way we didn't anticipate.
I already went through these steps: https://aws.amazon.com/premiumsupport/knowledge-center/configure-cloudfront-to-forward-headers/ but ended up cutting the connection between the API and the frontend.
We don't care about caching implications (we don't have a lot of traffic), we just need to the right fields to show up in the backend.
By default, most request headers are removed, because CloudFront's default behaviors are generally designed around optimal caching. CloudFront's default header handling behavior is documented.
If you need to see specific headers at the origin, whitelist those headers for forwarding in the cache distribution. The documentation refers to this as “Selecting the Headers on Which You Want CloudFront to Base Caching” -- and that is what it does -- but that description masks what's actually happening. CloudFront removes the rest of the headers because it has no way of knowing for certain whether a specific header with a certain value might change the response that the origin generates. If it didn't remove these headers by default, there would be confusion in the other direction when users saw the "wrong" responses served from the cache.
In your case, you almost certainly don't want to include the Host header in what you are whitelisting for forwarding.
When testing, especially, be sure you also set the Error Caching Minimum TTL to 0, because the default value is 300 seconds... so you can't see whether the problem is fixed for up to 5 minutes after you fix it. This default is also by design, a protective measure to avoid overloading your origin with requests that are likely to continue to fail.
When examining responses from CloudFront, keep an eye on the Age response header, which is present any time the response is served from cache. It tells you how long it's been (in seconds) since CloudFront obtained the response it is currently returning to you.
If you want to disable CloudFront caching, you can set Maximum, Minimum, and Default TTL all to 0 (this only affects 2xx and 3xx HTTP responses -- errors are cached for a different time window, as noted above), or your origin can consistently return Cache-Control: s-maxage=0, which will prevent CloudFront from caching the response.
I am thinking to use cloudflare to cache a resource generated by a REST API endpoint.
Because the API can take time to return the result, I am wondering if it is possible to configure cloudflare to refresh the resource in background returning always the cached resource to clients.
You can use page rules on the API endpoint to cache the result for X hours (or days, etc).
I think it will have to be GET though, I don't think POST is ever cached.
I'm investigating a problem a user is having with a web application that is built using Yii.
The user is not seeing the Yii 'flash' session-based user-feedback messages. These messages are shown once to a user and then destroyed (so they're not shown on subsequent page loads).
I took a look at the server access logs and I noticed something weird.
When this user requests a page there is a second identical request but from a different IP and with a different User Agent string. The second request is often at the same time or is sometimes (at most) a couple of minutes later. A bit of googling leads me to the conclusion that the user is browsing the web using a HTTP Proxy.
So, is this likely to be a HTTP Proxy? Or could it be something more suspicious? And if it is a HTTP Proxy, does this explain why they're not seeing the flash session messages? Could it be that the messages are being 'shown' to the Proxy and then destroyed?
We have recently fixed a nagging error on our website similar to the one described in How to stop javascript injection from vodafone proxy? - basically, the Vodafone mobile network was vandalizing our pages in transit, making edits to the JavaScript which broke viewmodels.
Adding a "Cache-Control: no-transform" header to the page that was experiencing the problem fixed it, which is great.
However, we are concerned that as we do more client-side development using JavaScript MVP techniques, we may see it again.
Is there any reason not to add this header to every page served up by our site?
Are there any useful transformations that this will prevent? Or is it basically just similar examples of carriers making ham-fisted attempts to minify things and potentially breaking them in the process?
The reasons not to add this header is speed performance and data transfer.
Some proxy / CDN services encode the media, so if your client is behind proxy or are you using a CDN service, the client may get higher speed and spend littler data transfer. This header actually orders proxy / CDN - not to encode the media , and leave the data as is.
So, if you don't care about this, or your app not use many files like images or music, or you don't want any encoding on your traffic, there is no reason not to do this (and the opposite, recommended to).
See the RFC here: http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.9.5
Google has recently incorporated the service googleweblight so if your pages has the "Cache-Control: no-transform" header directive you'll be opting-out from transcoding your page in case the connection comes from a mobile device with slow internet connection.
More info here:
https://support.google.com/webmasters/answer/6211428?hl=en