Service Worker Response Cache Headers - browser-cache

I'm trying to figure out how the service worker works in regards to cache headers in responses. I have implemented a couple of service workers now but have never had to bother worrying about caching headers, how long items should be cached for etc. I'm now implementing it on an enterprise production site, whereby this stuff actually really matters.
Basically when using a service worker, is the http cache completely bypassed?
Do I then need to build a framework to handle resource expiration/invalidation like the http cache used to do for us? Or am I talking rubbish?
Would be super helpful if someone could provide some clarification of this. The way I see it there are 3 potential scenarios:
A). Network request => Service worker fetch => (Browser cache?) <=> Server
B). Network request <=> (Browser cache?) <=> Service worker fetch <=> Server
C). Network request => Service worker fetch <=> Server
I've tested this locally and it seems that C). is the correct implementation, whereby we the developer have sacrificed cache header/duration abstraction for control.
I'm fine with this, just want it clarifying before I run off and build a framework for reading and honouring caching headers in the service worker.

A) is the correct model. If a service worker controls a page, all network requests will trigger the fetch event handler of the service worker prior to consulting the browser cache or the network.
In turn, any time the service worker makes a network request, either explicitly via fetch() or implicitly via cache.add()/cache.addAll(), the browser's "traditional" cache is consulted for a response first. A network request to the server will only be made if there isn't a valid response in the browser cache.
This sometimes works in your favor, and sometimes can expose you to subtle bugs if you don't expect that behavior.
There's a very detailed explanation of this interaction at https://jakearchibald.com/2016/caching-best-practices/, specifically covering things to avoid and ways to take advantage of this behavior.

That depends on how you configure request. With Fetch API you can control how request interacts with browser HTTP Cache.
For example you can set request's cache mode to no-store so it will bypass HTTP Cache. Or you can set request's cache mode to force-cache so browser will return cached response even if it is stale:
fetch("some.json", {cache: "force-cache"})
.then(function(response) { /* consume the response */ });
By default request's cache mode is default. In this case request will act as usual. Obviously that is only if service worker actually performs request instead of returning some hardcoded response.
For more information check Fetch Standard, Request.cache MSN page and Using Service Workers MDN page.

Related

How to use Chrome DevTools protocol in Selenium (using Python) for capturing HTTP requests and responses?

I know that Fetch Domain is used for this purpose but I do not know how exactly I can implement it. In Selenium python, I used the following code to enable issuing of requestPaused events.
driver.execute_cdp_cmd("Fetch.enable",{})
driver.get('https://www.example.com')
But I do not know how can I handle requestPaused event (I need to call one fulfillRequest or continueRequest/continueWithAuth). As a result, my program stops working.
I really appreciate it if anyone could provide me an example to help me understand how it works.
Yes, you saw it right.
As per the release notes of Selenium v4.0.0-alpha-3:
* Expose devtools APIs from chromium derived drivers.
* Expose presence of devtools support on a role-based interface
As per the release notes of Selenium v4.0.0.0-alpha-1:
* Basic support for CDP landed via the "DevTools" interface.
So chrome-devtools-protocol is all set to be available with selenium4 which will allow for tools to instrument, inspect, debug and profile Chromium, Chrome and other Blink-based browsers. In the discussion Controlling Chrome Devtools with Selenium Webdriver #AdiOhana mentions of the example usage of a few commands from the Profiler Domain as follows:
driver.getDevTools().createSession();
driver.getDevTools().send(new Command("Profiler.enable", ImmutableMap.of()));
driver.getDevTools().send(new Command("Profiler.start", ImmutableMap.of()));
//register to profiler events
driver.getDevTools().addListener(new Event("Profiler.consoleProfileStarted", ConsoleProfileStarted.class), new Consumer<Object>() {
#Override
public void accept(Object o) {
//do something
}
});
Note: Until the Profiler domain will added to Selenium java client, you will have to supply your Mapper.
Fetch Domain
Fetch Domain will enable clients substitute browser's network layer with client code.
The Fetch Domain methods are as follows:
Fetch.disable: Disables the fetch domain.
Fetch.enable: Enables issuing of requestPaused events. A request will be paused until client calls one of failRequest, fulfillRequest or continueRequest/continueWithAuth.
Fetch.failRequest: Causes the request to fail with specified reason.
Fetch.fulfillRequest: Provides response to the request.
Fetch.continueRequest: Continues the request, optionally modifying some of its parameters.
Fetch.continueWithAuth: Continues a request supplying authChallengeResponse following authRequired event.
Fetch.getResponseBody: Causes the body of the response to be received from the server and returned as a single string. May only be issued for a request that is paused in the Response stage and is mutually exclusive with takeResponseBodyForInterceptionAsStream. Calling other methods that affect the request or disabling fetch domain before body is received results in an undefined behavior.
Fetch.takeResponseBodyAsStream: Returns a handle to the stream representing the response body. The request must be paused in the HeadersReceived stage. Note that after this command the request can't be continued as is -- client either needs to cancel it or to provide the response body. The stream only supports sequential read, IO.read will fail if the position is specified. This method is mutually exclusive with getResponseBody. Calling other methods that affect the request or disabling fetch domain before body is received results in an undefined behavior.
The Fetch Domain events are as follows:
Fetch.requestPaused: Issued when the domain is enabled and the request URL matches the specified filter. The request is paused until the client responds with one of continueRequest, failRequest or fulfillRequest. The stage of the request can be determined by presence of responseErrorReason and responseStatusCode -- the request is at the response stage if either of these fields is present and in the request stage otherwise.
Fetch.authRequired: Issued when the domain is enabled with handleAuthRequests set to true. The request is paused until client responds with continueWithAuth.
References
You can find a couple of revelant discussions in:
Can Selenium WebDriver (java) interact with the Browser's inspect tool element selector?
What is the difference between WebDriver and DevTool protocol

Trying to set up distributed event tracing

I am trying to set up distributed event tracing throughout out microservice architecture.
Here is some preamble about our architecture:
Traefik load balancer that forwards request to the appropriate backend service based on the route pathname.
Frontend application on a "catchall" route that is served whenever a route is not caught by another microservice.
Various backend services in node/dotnetcore listening on /api/<serviceName>
traefik is setup with the traceContextHeaderName set to "trace-id".
How I imagine this would work is that the frontend application receives a header "trace-id" from the load balancer with a value that can be used to "link" the spans together for requests that are related.
Example scenario:
When a customer loads attempts to sign in, they make a request for the web application, receive and render the HTML/CSS/JS, then the subsequent requests to /api/auth/login can be POSTed with the login data and the value of the "trace-id" header supplied by traefik.
The backend service that handles the /api/auth/login endpoint can capture this "trace-id" header value and publish some spans to jaeger related to the work that it is doing to validate the user.
What is happening:
When the request is made for the frontend HTML, no "trace-id" header is received so any subsequent spans that are published are all considered individual traces and are not linked together.
traefik.toml:
...
[tracing]
backend = "jaeger"
serviceName = "traefik"
spanNameLimit = 0
[tracing.jaeger]
samplingServerURL = "http://jaeger:5778/sampling"
samplingType = "const"
samplingParam = 1.0
localAgentHostPort = "jaeger:6831"
traceContextHeaderName = "trace-id"
...
I understand that StackOverflow is not a "code it for me" service. I am looking for guidance on what could possibly be going wrong as I am new to distributed event tracing.
I have tried googling and searching for answers but I have come to a dead end.
Any help/suggestions on where to look would be greatly appreciated.
Please let me know if I am barking up the wrong tree, approaching this incorrectly, or if my understanding of how the traceContextHeaderName should work is incorrect.
Ugh. I am an idiot.
Here is what was going wrong for anyone else who might be stuck something like this:
The frontend application is receiving a header, I was just looking in the wrong place.
The request comes from the load balancer to the node frontend microservice which sends its response to the browser.
I was checking the browser for the header, but the node frontend microservice was not forwarding this header to the browser.

Solve 429 Too Many Requests issue while load testing using Jmeter

I'm trying to hit the server with multiple requests for load testing. I have setup JMeter thread with 100 concurrent users per seconds. Each request read data from CSV (different tokens to identify users). But after few request I'm getting error:
429 Too Many Requests
How we can solve this? Any settings in Jmeter?
Or Do I need to change Apache settings?
As per HTTP Status 429 documentation
The HTTP 429 Too Many Requests response status code indicates the user has sent too many requests in a given amount of time ("rate limiting").
A Retry-After header might be included to this response indicating how long to wait before making a new request.
Most probably your JMeter test configuration is not correct, the reasons could be in:
You don't use HTTP Cookie Manager so all the requests fall under the same session and application reacts correspondingly
Your application limits rate of requests from a single IP. Consider going for IP Spoofing or Distributed Testing or both
You're using the same credentials for all virtual users, make sure that each JMeter virtual user has its own credentials and operates its own data. You can use i.e. CSV Data Set Config for parameterization.
This is most probably a DDOS protection in the application you’re testing so your options are to ask the infrastructure team or developers to disable it.
As per reference documentation:
The HTTP 429 Too Many Requests response status code indicates the user has sent too many requests in a given amount of time ("rate limiting").
A Retry-After header might be included to this response indicating how long to wait before making a new request.
There is nothing you can do ok jmeter side.

JMeter's ResourceLastAccessedTime GET request failed

I am trying to use JMeter to test our Web Application. We originally used LoadComplete to test our Web Application, but because LoadComplete is not able to run on a non-GUI mode, we were not able to use the max stat’s from our test server (strain our 8 CPU’s and 8GB’s of RAM). That is why I moving towards JMeter (https://blazemeter.com/blog/5-ways-launch-jmeter-test-without-using-jmeter-gui).
The test includes logging in, choosing a specific app, do a simple task through this app and then end the recording. The HTTP Requests, which are failing are printing Failed Access on their Response Data on the View Results Tree.
I used the HTTP(S) Test Script Recorder to record each HTTP request. My JMeter project is failing on a few different HTTP Requests, which includes oauthtoken Get Request that includes jessionid="item", a GET resourceLastAccessedTime Request, and a couple GET resourceLastAccessedTime Requests. I tried to follow blazementer's guidance for how to use JMeter for Login Authentication, because these requests seem to be involved with the authentication of each user after logging in and the problem I am getting on Response data for each of these requests on the View Results Tree it says Access denied. (https://docs.blazemeter.com/customer/portal/articles/1743663-how-to-use-jmeter-for-login-authentication-).
One of the steps is to "copy and paste" the Parameters from the Post request after you login to these requests. I can add these parameters to these requests right below where it says Send Parameters with the request, but our POST request only has two parameters (the login name and the password). Is there somewhere else to look for these parameters?
I tried a combination of a lot of different attempts, but I am still unsuccessful (meaning: I moved the Regular Expression Extractor to a few different HTTP requests and I moved which HTTP requests to put those parameters and I have not been successful yet).
Do you know of a URL that could be helpful for this?
Don't trust Test Script Recorder! It doesn't follow any logic while recording your requests. It just records requests processed through proxy as they are. In case you use parameters that can't be defined as constants, the best way would be to rewrite the script manually.
Be patient and spend some hours (only once!) to learn how to construct any test scenarios (even complex) manually using Jmeter GUI. It will save you a lot of time for debugging.
It seems like (just a guess) that your test scenario doesn't contain Cookie Manager item. Based on what you wrote above, it seems like after logging to serer (by sending POST with login and password) it sets some cookies by Set-Cookie HTTP header. These cookies should be included in every next request as a prove that you successfully logged in before (the most common logic for simple web applications). So, if you get Access Denied, means you didn't include appropriate cookies in test request. Use Cookie Manager for that.
Feel free to ping me in case you need any assistance.
Jmeter help manual is all you need to know about how each element works.
P.S.: Jmeter also can generate distributed load from multiple slave servers, in GUI and CLI modes both. So, in case you need to stress your server yout, Jmeter is the best choise.
And welcome to Jmeter users family! Good luck.

Strategies to block an external webservice to simulate "down" during for a testing scenario?

I am working to integrate data from an external web service in the client side of my appliction. Someone asked me to test the condition when the service is unavailable or down. Anyone have any tips on how to block this site temporarily while we run the test to see how the service degrades?
For those curious we are testing against Virtual Earth, but Google Maps but this would apply to any equally complicated external service.
any thoughts and suggestions are welcome
Create some Mock-Webservice class or interface (and inject it). In there, you could test the response of your system to webservice failures and also what happens, if a web-service request take longer than expected or actually time-out.
DeveloperWorks article on mock testing: http://www.ibm.com/developerworks/library/j-mocktest.html
You need to be sure to test the most common failure modes for this:
DNS lookup fails
IP connection fails (once DNS lookup succeeds)
HTTP response other than 200
HTTP response incomplete or timeout
HTTP response 200 but RPC or document returned is invalid
Those are just a few common failure modes I could think of that will all manifest themselves with different behaviors that you may wish to have your application handle explicitly.
If you set up a computer between the caller and service that routes between them, you can simulate each of these failure modes distinctly and modify your application to handle them.
How about blocking the domain name(s) in question by putting a nonsense entry into the hosts file?