When using the webdriver to start a web page, can you intercept the relevant requests of the web page to obtain the request payload?
Selenium 4 have network intercept capablity
Manipulating network traffic is extremely useful for testing web applications because it gives us endless possibilities to configure the environment where the web application will be running. Blocking network traffic, mocking network connections, simulating the network speed, adding headers, etc. The following code shows how to add a header to the HTTP request, which is helpful when our application under test exposes filters requests or exposes specific features depending on the received headers.
#Test
public void addExtraHeaders() {
ChromeDriver driver = new ChromeDriver();
DevTools devTools = driver.getDevTools();
devTools.createSession();
driver.get("https://manytools.org/http-html-text/http-request-headers/");
devTools.send(Network.enable(Optional.empty(), Optional.empty(), Optional.empty()));
Headers headers = new Headers(Collections.singletonMap("testing", "selenium"));
devTools.send(Network.setExtraHTTPHeaders(headers));
driver.get("https://manytools.org/http-html-text/http-request-headers/");
driver.quit();
}
Reference : https://saucelabs.com/resources/articles/bidirectional-apis
Related
I'd like to know how can I put the HTTP header manager in JMeter if I use Selenium Webdriver Sampler.
I know that there is the standard tool(HTTP Header Manager) in JMeter but that tool is useful when I use HTTP Request in my test. In this case for testing, I use only WebDriver Sampler with Java 1.8. The goal is to see in dynatrace the tags that I send from JMeter. Is it possible to do that? And if it is the answer is positive, how can I do that? Thanks for your help!
WebDriver Sampler doesn't respect HTTP Header Manager
WebDriver itself doesn't support working with HTTP Headers and the feature is unlikely to be implemented ever
So the options are in:
Use an extension like ModHeader, but in this case you will have so switch from the WebDriver Sampler to JSR223 Sampler. Example code:
def options = new org.openqa.selenium.chrome.ChromeOptions()
options.addExtensions(new File('/path/to/modheaders.crx'))
def capabilities = new org.openqa.selenium.remote.DesiredCapabilities()
capabilities.setCapability(org.openqa.selenium.chrome.ChromeOptions.CAPABILITY, options)
def driver = new org.openqa.selenium.chrome.ChromeDriver(capabilities)
driver.get('http://example.com')
Use a proxy like BrowserMob as the proxy for the WebDriver and configure it to add headers to each intercepted request. Example initialization code (you can put it into the aformentioned JSR223 Sampler somewhere in setUp Thread Group)
def proxy = new net.lightbody.bmp.BrowserMobProxyServer()
def proxyPort = 8080
proxy.setTrustAllServers(true)
proxy.addRequestFilter((request, contents, info) -> {
request.headers().add('your header name', 'your header value')
return null
})
proxy.start(proxyPort)
I know that Fetch Domain is used for this purpose but I do not know how exactly I can implement it. In Selenium python, I used the following code to enable issuing of requestPaused events.
driver.execute_cdp_cmd("Fetch.enable",{})
driver.get('https://www.example.com')
But I do not know how can I handle requestPaused event (I need to call one fulfillRequest or continueRequest/continueWithAuth). As a result, my program stops working.
I really appreciate it if anyone could provide me an example to help me understand how it works.
Yes, you saw it right.
As per the release notes of Selenium v4.0.0-alpha-3:
* Expose devtools APIs from chromium derived drivers.
* Expose presence of devtools support on a role-based interface
As per the release notes of Selenium v4.0.0.0-alpha-1:
* Basic support for CDP landed via the "DevTools" interface.
So chrome-devtools-protocol is all set to be available with selenium4 which will allow for tools to instrument, inspect, debug and profile Chromium, Chrome and other Blink-based browsers. In the discussion Controlling Chrome Devtools with Selenium Webdriver #AdiOhana mentions of the example usage of a few commands from the Profiler Domain as follows:
driver.getDevTools().createSession();
driver.getDevTools().send(new Command("Profiler.enable", ImmutableMap.of()));
driver.getDevTools().send(new Command("Profiler.start", ImmutableMap.of()));
//register to profiler events
driver.getDevTools().addListener(new Event("Profiler.consoleProfileStarted", ConsoleProfileStarted.class), new Consumer<Object>() {
#Override
public void accept(Object o) {
//do something
}
});
Note: Until the Profiler domain will added to Selenium java client, you will have to supply your Mapper.
Fetch Domain
Fetch Domain will enable clients substitute browser's network layer with client code.
The Fetch Domain methods are as follows:
Fetch.disable: Disables the fetch domain.
Fetch.enable: Enables issuing of requestPaused events. A request will be paused until client calls one of failRequest, fulfillRequest or continueRequest/continueWithAuth.
Fetch.failRequest: Causes the request to fail with specified reason.
Fetch.fulfillRequest: Provides response to the request.
Fetch.continueRequest: Continues the request, optionally modifying some of its parameters.
Fetch.continueWithAuth: Continues a request supplying authChallengeResponse following authRequired event.
Fetch.getResponseBody: Causes the body of the response to be received from the server and returned as a single string. May only be issued for a request that is paused in the Response stage and is mutually exclusive with takeResponseBodyForInterceptionAsStream. Calling other methods that affect the request or disabling fetch domain before body is received results in an undefined behavior.
Fetch.takeResponseBodyAsStream: Returns a handle to the stream representing the response body. The request must be paused in the HeadersReceived stage. Note that after this command the request can't be continued as is -- client either needs to cancel it or to provide the response body. The stream only supports sequential read, IO.read will fail if the position is specified. This method is mutually exclusive with getResponseBody. Calling other methods that affect the request or disabling fetch domain before body is received results in an undefined behavior.
The Fetch Domain events are as follows:
Fetch.requestPaused: Issued when the domain is enabled and the request URL matches the specified filter. The request is paused until the client responds with one of continueRequest, failRequest or fulfillRequest. The stage of the request can be determined by presence of responseErrorReason and responseStatusCode -- the request is at the response stage if either of these fields is present and in the request stage otherwise.
Fetch.authRequired: Issued when the domain is enabled with handleAuthRequests set to true. The request is paused until client responds with continueWithAuth.
References
You can find a couple of revelant discussions in:
Can Selenium WebDriver (java) interact with the Browser's inspect tool element selector?
What is the difference between WebDriver and DevTool protocol
I'm looking for a way in Chrome DevTools (or any equivalent) to control the HTTP requests done by my web application:
I want to approve HTTP requests before executing, or let them fail in an unexpected way (give it status 500 or something).
USAGE EXAMPLE: Testing unexpected behavior
Does anyone know a way to achieve this.
I see 2 possible solutions to achieve this goal on the client side:
Use the Request-Blocking panel from the Drawer (Open Chrome DevTools -> Esc -> '...' -> Request blocking
This is completely out-of-the-box and works for most "offline-first" use cases.
Use a service worker. They are basically a way to proxy requests and respond individually (e.g. by responding with a 500-er). You might want to enable/disable such a debugging-feature by using Chrome Devtools Snippets (Open Chrome DevTools -> Sources -> Snippets) as you don't want your requests to fail all the time :)
First you need to register your serviceworker like this:
if('serviceWorker' in navigator) {
navigator.serviceWorker.register('/path-to-service-worker.js').then(function(registration) {
// registration successful
}).catch(function(err) {
// registration failed
});
}
Afterwards reload the browser (or install your service-worker in the DevTools -> Application -> Service Workers) so that your service-worker.js is active, may listen to the 'fetch' event and proxy the requests for this domain like this:
self.addEventListener('fetch', function(event) {
// this will set a breakpoint in chrome devtools, allowing you to manually edit the response
debugger;
// alternatively you could reponse with an error response like this:
event.respondWith(
new Response(null, {
status: 500
})
);
});
Sidenote: Due to security restrictions in the browser serviceworkers only work over https and on localhost.
Further information:
https://developer.mozilla.org/en-US/docs/Web/API/Response/Response
https://developers.google.com/web/fundamentals/primers/service-workers/
You can use Requestly Chrome extension to Redirect, Cancel, Block, Modify headers,... of the requests.
To approve requests before executing e.g. for AJAX requests create a redirect rule and point it to a static JSON file or another script.
To block a request use cancel request feature and set a custom pattern.
How does selenium commands browser ?
I know that selenium proxy's requests to urls and on reponse injects selenium core js api into the response. but how does "click" or any other command reach the browser ? In other words how does selenium invoke the selenium core api injected in the browser page ?
Selenium recommends the use of WebDriver to replace the remote control functionality. TMK, the older code injected Javascript into the browser and communicated via that. The WebDriver code uses the native API for each browser.
"Selenium-WebDriver makes direct calls to the browser using each browser’s native support for
automation." via http://seleniumhq.org/docs/03_webdriver.html#selenium-2-0-features
Selenium RC runs a request-loop in the browser (in the Selenium window or frame), and the RC server acts as a forwarding agent between the client and the browser. Here's the sequence:
The test script calls Selenium.Click('SUBMIT').
The client issues an HTTP request that tells the server "next time the browser asks, tell it to click on the 'SUBMIT' button".
The browser may already have an HTTP request waiting for a response, but if not, it will soon.
When the browser sends its HTTP request, the server responds with "click the 'SUBMIT' button".
The browser does what it's told and sends another HTTP request to the server, indicating that the command succeeded.
The server responds to the client's existing HTTP request with an "OK" response.
The Click() routine returns to the script.
I'm using Stream hub as my push server.
when i used Firefox 4.0.1 as my browser, i saw that stream hub is making periodical Ajax request. But in safari i noticed an error in console regarding ws: (web socket), so i came to a conclusion that safari is making request using ws:// url's while Firefox is making plain Ajax calls.
this is my client code
function start() {
hub = new StreamHub();
hub.connect("http://localhost:7878/streamhub/");
hub.subscribe("HelloWorld", topicUpdated);
console.log("brws. subscribed to helloworld");
}
I have two questions.
1)Is my conclusion correct?
2)how do i monitor HTTP request and response in safari (like firebug net panel).
Thanks