How can I read an XHR response and parse it in Katalon Studio?
I currently use a workaround way of testing responsiveness of my app: I use various waitForElement_*_() (*=visible, clickable, present, not-visible, not-clickable, not-present) commands in order to measure loading time of various elements.
I would like to get more specific and measure the duration of network requests (that can be seen in DevTools - network traffic).
Can it be done?
I am not sure if it can be done using Katalon studio. I am replying to your post, because I use network traffic information to derive performance numbers, and I use browsermobproxy.
Needless to say, this reply does not answer your question, just an option of using browsermobproxy
How to access the values of Chrome's Dev tools Network tab's Request or summary using Selenium in python/java?
In Katalon 7 and with and with Chrome DevTools Protocol Integration plugin, as was described here you can intercept network requests.
The following example shows how to mock search requests in Wikipedia so that the result will always be “Katalon Studio”.
import static com.kms.katalon.core.testobject.ObjectRepository.findTestObject
import com.github.kklisura.cdt.protocol.commands.Fetch as Fetch
import com.github.kklisura.cdt.protocol.commands.Page as Page
import com.github.kklisura.cdt.services.ChromeDevToolsService as ChromeDevToolsService
import com.katalon.cdp.CdpUtils as CdpUtils
import com.kms.katalon.core.util.internal.Base64 as Base64
import com.kms.katalon.core.webui.keyword.WebUiBuiltInKeywords as WebUI
import com.kms.katalon.core.testobject.ConditionType
import com.kms.katalon.core.testobject.TestObject as TestObject
WebUI.openBrowser('')
ChromeDevToolsService cdts = CdpUtils.getService()
Page page = cdts.getPage()
Fetch fetch = cdts.getFetch()
fetch.onRequestPaused({ def requestIntercepted ->
String interceptionId = requestIntercepted.getRequestId()
String url = requestIntercepted.getRequest().getUrl()
boolean isMocked = url.contains('api.php')
String response = '["Katalon Studio",["Katalon Studio"],["Katalon Studio is an automation testing solution developed by Katalon LLC."],["https://en.wikipedia.org/wiki/Katalon_Studio"]]'
String rawResponse = Base64.encode(response)
System.out.printf('%s - %s%s', isMocked ? 'MOCKED' : 'CONTINUE', url, System.lineSeparator())
if (isMocked) {
fetch.fulfillRequest(interceptionId, 200, new ArrayList(), rawResponse, null)
} else {
fetch.continueRequest(interceptionId)
}
})
fetch.enable()
page.enable()
WebUI.navigateToUrl('https://en.wikipedia.org/wiki/Main_Page')
TestObject searchInput = new TestObject().addProperty('css', ConditionType.EQUALS, '#searchInput')
TestObject containing = new TestObject().addProperty('xpath', ConditionType.EQUALS, "//div[div[contains(.,'containing...')]]")
WebUI.setText(searchInput, 'Intercept request')
WebUI.waitForElementVisible(containing, 10)
NOTES:
Original post on Katalon forum: https://forum.katalon.com/t/intercepting-request-with-chrome-devtools-protocol/36081.
Sample project used in this topic: https://github.com/katalon-studio-samples/katalon-studio-chrome-devtools-protocol-plugin-samples.
The plugin uses https://github.com/kklisura/chrome-devtools-java-client to connect to CDP.
Related
Trying an example to hit a rest API , but seeing a 400 status code. Is this the correct way to call an API using form params?
import javax.json.JsonObject;
import javax.ws.rs.client.Client;
import javax.ws.rs.client.ClientBuilder;
import javax.ws.rs.core.Form;
import javax.ws.rs.client.Entity;
import javax.ws.rs.client.WebTarget;
import javax.ws.rs.client.Invocation;
import javax.ws.rs.HttpMethod;
import javax.ws.rs.core.Response;
...
public JsonObject getUserProfile() throws Exception {
Form userform = new Form();
userform.param("grant_type", "password")
.param("client_id", "cexxx")
.param("username", "theuser")
.param("password", "password")
.param("scope", "user.read openid profile offline_access");
Client client = ClientBuilder.newClient();
String serverUrl = "https://login.microsoftonline.com/547xx/oauth2/v2.0/token";
WebTarget target = client.target(serverUrl);
final Invocation.Builder invocationBuilder = target.request();
invocationBuilder.header("Content-Type", "application/x-www-form-urlencoded");
final Response response = invocationBuilder.method(
HttpMethod.POST,
Entity.entity(userform, MediaType.APPLICATION_FORM_URLENCODED),
Response.class);
System.out.println("r.getStatus()=" + response.getStatus());
...
}
The same works on Postman:
Thanks Giorgi for the hint.
Actually, the code we have above to programmatically make an API call works. The issue is that we are seeing error from server side.
Using this, we are able to see the error message from server:
System.out.println(response.readEntity(String.class));
{"error":"invalid_grant","error_description":"AADSTS50034: The user account sx does not exist in the 547..b32 directory. To sign into this application, the account must be added to the directory.\r\nTrace ID: 4616...3c00\r\nCorrelation ID: 617...a01\r\nTimestamp: 2020-09-29 22:25:41Z","error_codes":[50034],"timestamp":"2020-09-29 22:25:41Z","trace_id":"461...c00","correlation_id":"617..a01","error_uri":"https://login.microsoftonline.com/error?code=50034"}
I want to use Python to fill this form.
I tried using Mechanize but this is a Microsoft Form which uses JavaScript and has no form tag and no GET/POST URL. Maybe BeautifulSoup/Selenium can do this, but I do not have any experience in scraping JS forms. Can anyone help me out and suggest how to go about this?
Here's what I've tried, Mechanize is unable to recognize any form on the page:
import mechanize
def main():
br = mechanize.Browser()
br.set_handle_robots(False)
br.set_handle_refresh(False)
br.addheaders = [('User-agent', 'Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.1) Gecko/2008071615 Fedora/3.0.1-1.fc9 Firefox/3.0.1')]
response = br.open("https://forms.office.com/Pages/ResponsePage.aspx?id=8Pm7rtoj40mYvzIXGrvJvCxQDveyljlCrKN2Teo3EHFUQVNaWDlYRkhYR09JRTZWRFpKTTNIQU9HUC4u")
for form in br.forms():
print("Form name:", form.name) #prints nothing
print(form) #prints nothing
if __name__ == '__main__':
main()
Selenium works fine.
You'll need to install the components
install selenium pip install selenium
You need to ensure you download the correct chromedriver (or other driver) for your browser and OS versions and add it to path
Then this runs:
from selenium import webdriver
driver = webdriver.Chrome()
url = "https://forms.office.com/Pages/ResponsePage.aspx?id=8Pm7rtoj40mYvzIXGrvJvCxQDveyljlCrKN2Teo3EHFUQVNaWDlYRkhYR09JRTZWRFpKTTNIQU9HUC4u"
driver.get(url)
name = driver.find_element_by_xpath("//div[#class='question-title-box'][.//span[text()='NAME']]/following-sibling::*//input")
name.send_keys("hello, World")
setionSelection = "F"
section = driver.find_element_by_xpath("//div[#class='question-title-box'][.//span[text()='Section']]/following-sibling::*//input[#value='" + setionSelection + "']")
section.click()
date = driver.find_element_by_xpath("//input[contains(#placeholder, 'Please input date')]")
date.send_keys("01/12/2020")
submit = driver.find_element_by_xpath("//div[text()='Submit']")
submit.click()
The xapths are a little long but they're based on the question text so potentially stable
For an alternative approach - When you say there is no POST url, did you check devtools? - That exposes the destination of the form:
Request URL: https://forms.office.com/formapi/api/aebbf9f0-23da-49e3-98bf-32171abbc9bc/users/f70e502c-96b2-4239-aca3-764dea371071/forms('8Pm7rtoj40mYvzIXGrvJvCxQDveyljlCrKN2Teo3EHFUQVNaWDlYRkhYR09JRTZWRFpKTTNIQU9HUC4u')/responses
Request Method: POST
it also exposes the payload... This is the first submit:
{startDate: "2020-08-17T10:40:18.504Z", submitDate: "2020-08-17T10:40:18.507Z",…}
answers: "[{"questionId":"r8f09d63e6f6f42feb2f8f4f8ed3f9389","answer1":"Hello, World"},{"questionId":"r28fe12073dfa47399f8ce95ae679dccf","answer1":"G"},{"questionId":"r8f9e9fedcc2e410c80bfa1e0e3ef9750","answer1":"2020-08-28"}]"
startDate: "2020-08-17T10:40:18.504Z"
submitDate: "2020-08-17T10:40:18.507Z"
Those post URL UUID/GUIDs questions IDs seem to be satic for this form. Every time i run form they're not chaning. This is the second run:
{startDate: "2020-08-17T10:43:48.544Z", submitDate: "2020-08-17T10:43:48.546Z",…}
answers: "[{"questionId":"r8f09d63e6f6f42feb2f8f4f8ed3f9389","answer1":"test me"},{"questionId":"r28fe12073dfa47399f8ce95ae679dccf","answer1":"G"},{"questionId":"r8f9e9fedcc2e410c80bfa1e0e3ef9750","answer1":"2020-08-12"}]"
startDate: "2020-08-17T10:43:48.544Z"
submitDate: "2020-08-17T10:43:48.546Z"
Once you capture this once you'll probably be able to do it through the API without a GUI.
... Just to make sure, i tried it and i get success...
import requests
url = "https://forms.office.com/formapi/api/aebbf9f0-23da-49e3-98bf-32171abbc9bc/users/f70e502c-96b2-4239-aca3-764dea371071/forms('8Pm7rtoj40mYvzIXGrvJvCxQDveyljlCrKN2Teo3EHFUQVNaWDlYRkhYR09JRTZWRFpKTTNIQU9HUC4u')/responses"
myobj = {"startDate":"2020-08-17T10:48:40.118Z","submitDate":"2020-08-17T10:48:40.121Z","answers":"[{\"questionId\":\"r8f09d63e6f6f42feb2f8f4f8ed3f9389\",\"answer1\":\"Hello again, World\"},{\"questionId\":\"r28fe12073dfa47399f8ce95ae679dccf\",\"answer1\":\"F\"},{\"questionId\":\"r8f9e9fedcc2e410c80bfa1e0e3ef9750\",\"answer1\":\"2020-08-26\"}]"}
x = requests.post(url, data = myobj)
My answers are just hard coded into the data object but it seems to work.
Remember to pip install requests if you don't already have it
I have a main feature file where I have included a "setup" feature file that should add some test data. This setup feature file has an annotation that I have called #ignore. However, following the instructions in this Can't be enable to #ignore annotation for the features SO answer, but I am still seeing the setup feature file being run outside of the main test feature.
Main feature file, unsubscribe_user.feature:
Feature: Unsubscribe User
Background:
* def props = read('properties/user-properties.json')
* url urlBase
* configure headers = props.headers
* def authoriZation = call read('classpath:basic-auth.js') { username: 'admin', password: 'admin' }
* def testDataSetup = call read('classpath:com/meanwhileinhell/app/karate/feature/mockserver/testDataSetup.feature') { data1: #(props.data1), data2: #(props.data2) }
Scenario: Unsubscribe user
...
...
Scenario: Remove test data
* def testDataTearDown = call read('classpath:com/meanwhileinhell/app/karate/feature/mockserver/testDataTearDown.feature') { data1: #(props.data1), data2: #(props.data2) }
...
testDataSetup.feature file
#ignore
Feature: Add data to REST Mock Server
Background:
* url mockServerUrlBase
Scenario: Add data
* print 'Adding test data'
Given path 'mapping'
And request { data1: '#(data1)', data2: '#(data2)' }
When method post
Then status 201
Now from my Java runner class, I have added #KarateOptions(tags = "~#ignore").
import org.junit.runner.RunWith;
import com.intuit.karate.KarateOptions;
import com.intuit.karate.junit4.Karate;
import cucumber.api.CucumberOptions;
#RunWith(Karate.class)
#CucumberOptions(features = "classpath:com/meanwhileinhell/app/karate/feature/unsubscribe_user.feature")
#KarateOptions(tags = "~#ignore")
public class KarateTestUnSubscribeUserRunner {
}
However, I can still see my print statement in my setup class being called, and two POSTs being performed. I have also tried running my suite with the following cmd options, but again, still see the feature file run twice.
./gradlew clean test -Dkarate.env=local -Dkarate.options="--tags ~#ignore" --debug
I am following this wrong somewhere? Is there something I can add to my karate-config.js file? I am using Karate version 0.9.0.
Annotations only work on the "top level" feature. Not on "called" features.
If your problem is that the features are being run even when not expected, you must be missing something, or some Java class is running without knowing it. So please follow this process and we can fix it: https://github.com/intuit/karate/wiki/How-to-Submit-an-Issue
EDIT: I think I got it - please don't mix CucumberOptions, in fact we deprecated it, use only KarateOptions. Even that is not recommended in 0.9.5 onwards and you should move to JUnit 5.
Read the docs: https://github.com/intuit/karate#karate-options
I have a jenkins job and I want to use it's lastSuccessfulBuild number in my Build Flow groovy script.
I can get the last successful build number from Jenkins api at:
http://{JENKINS_DOMAIN}/job/{JOB_NAME}/lastSuccessfulBuild/buildNumber
I tried using groovy's RESTClient in my Build Flow groovy script but when importing the groovyx.net.http.RESTClient library I get syntax error.
Does any one know away of getting around this error or getting the api result in some other way?
maybe this will help you:
import hudson.model.Build;
println(build.getProject().getLastSuccessfulBuild())
for example we have simple build flow groovy script building only one item "JobA". If we want check and print its last successful build we can write such script:
import hudson.model.Build;
def buildA = build("jobA")
println(buildA.getProject().getLastSuccessfulBuild())
Possibly a little overkill, but you can use HttpClient, as all you need is a get request on the url.
Here's one I knocked up from some code I had lying around
Tested it on our own Jenkins instance which has basic auth over ssl.
import org.apache.http.HttpResponse
import org.apache.http.HttpVersion
import org.apache.http.client.HttpClient
import org.apache.http.client.methods.HttpGet
import org.apache.http.client.params.ClientPNames
import org.apache.http.conn.ClientConnectionManager
import org.apache.http.conn.scheme.PlainSocketFactory
import org.apache.http.conn.scheme.Scheme
import org.apache.http.conn.scheme.SchemeRegistry
import org.apache.http.conn.ssl.SSLSocketFactory
import org.apache.http.impl.client.DefaultHttpClient
import org.apache.http.impl.conn.PoolingClientConnectionManager
import org.apache.http.params.BasicHttpParams
import org.apache.http.params.HttpConnectionParams
import org.apache.http.params.HttpParams
import org.apache.http.params.HttpProtocolParams
class LastSuccessfulBuild {
def static main(args) {
println new LastSuccessfulBuild().connect("your.jenkins.com", "443", "/path/to/job/YourJob/lastSuccessfulBuild/buildNumber", "your.user:your-password")
}
def connect(host, port, path, auth) {
def url = new URL("https", host, Integer.parseInt(port), path)
HttpClient client = createClient()
HttpGet get = new HttpGet(url.toURI())
get.setHeader("Authorization", "Basic ${auth.getBytes().encodeBase64().toString()}")
HttpResponse response = client.execute(get)
def status = response.statusLine.statusCode
if (status != 200) {
throw new IOException("Failed to get page, status: $response.statusLine")
}
return response.entity.content.text
}
def createClient() {
HttpParams params = new BasicHttpParams()
HttpProtocolParams.setVersion(params, HttpVersion.HTTP_1_1)
HttpProtocolParams.setContentCharset(params, "UTF-8")
params.setBooleanParameter(ClientPNames.HANDLE_REDIRECTS, true)
SchemeRegistry registry = new SchemeRegistry()
registry.register(new Scheme("http", PlainSocketFactory.getSocketFactory(), 80))
registry.register(new Scheme("https",SSLSocketFactory.getSocketFactory(),443))
ClientConnectionManager ccm = new PoolingClientConnectionManager(registry)
HttpConnectionParams.setConnectionTimeout(params, 8000)
HttpConnectionParams.setSoTimeout(params, 5400000)
HttpClient client = new DefaultHttpClient(ccm, params)
return client
}
}
Im working with the selenium remote driver to automate actions on a site, i can open the page i need directly by engineering the url as the sites url schema is very constant. This speeds up the script as it dose not have to work through several pages before it gets to the one it needs.
To make the automation seem organic is there a way to set a referral page in Selenium ?
If you're checking the referrer on the server, then using a proxy (as mentioned in other answers) will be the way to go.
However, if you need access to the referrer in Javascript using a proxy will not work. To set the Javascript referrer I did the following:
Go to the referral website
Inject this javascript onto the page via Selenium API: document.write('<script>window.location.href = "<my website>";</script>')"
I'm using a Python wrapper around selenium, so I cannot provide the function you need to inject the code in your language, but it should be easy to find.
What you are looking for is referer spoofing.
Selenium does not have an inbuilt method to do this, however it can be accomplished by using a proxy such as fiddler.
Fiddler also provides an API-only version of the FiddlerCore component, and programmatic access to all of the proxy's settings and data, thus allowing you to modify the headers of the http response.
Here is a solution in Python to do exactly that:
https://github.com/j-bennet/selenium-referer
I described the use case and the solution in the README. I think github repo won't go anywhere, but I'll quote the relevant pieces here just in case.
The solution uses libmproxy to implement a proxy server that only does one thing: adds a Referer header. Header is specified as command line parameter when running the proxy. Code:
# -*- coding: utf-8 -*-
"""
Proxy server to add a specified Referer: header to the request.
"""
from optparse import OptionParser
from libmproxy import controller, proxy
from libmproxy.proxy.server import ProxyServer
class RefererMaster(controller.Master):
"""
Adds a specified referer header to the request.
"""
def __init__(self, server, referer):
"""
Init the proxy master.
:param server: ProxyServer
:param referer: string
"""
controller.Master.__init__(self, server)
self.referer = referer
def run(self):
"""
Basic run method.
"""
try:
print('Running...')
return controller.Master.run(self)
except KeyboardInterrupt:
self.shutdown()
def handle_request(self, flow):
"""
Adds a Referer header.
"""
flow.request.headers['referer'] = [self.referer]
flow.reply()
def handle_response(self, flow):
"""
Does not do anything extra.
"""
flow.reply()
def start_proxy_server(port, referer):
"""
Start proxy server and return an instance.
:param port: int
:param referer: string
:return: RefererMaster
"""
config = proxy.ProxyConfig(port=port)
server = ProxyServer(config)
m = RefererMaster(server, referer)
m.run()
if __name__ == '__main__':
parser = OptionParser()
parser.add_option("-r", "--referer", dest="referer",
help="Referer URL.")
parser.add_option("-p", "--port", dest="port", type="int",
help="Port number (int) to run the server on.")
popts, pargs = parser.parse_args()
start_proxy_server(popts.port, popts.referer)
Then, in the setUp() method of the test, proxy server is started as an external process, using pexpect, and stopped in tearDown(). Method called proxy() returns proxy settings to configure Firefox driver with:
# -*- coding: utf-8 -*-
import os
import sys
import pexpect
import unittest
from selenium.webdriver.common.proxy import Proxy, ProxyType
import utils
class ProxyBase(unittest.TestCase):
"""
We have to use our own proxy server to set a Referer header, because Selenium does not
allow to interfere with request headers.
This is the base class. Change `proxy_referer` to set different referers.
"""
base_url = 'http://www.facebook.com'
proxy_server = None
proxy_address = '127.0.0.1'
proxy_port = 8888
proxy_referer = None
proxy_command = '{0} {1} --referer {2} --port {3}'
def setUp(self):
"""
Create the environment.
"""
print('\nSetting up.')
self.start_proxy()
self.driver = utils.create_driver(proxy=self.proxy())
def tearDown(self):
"""
Cleanup the environment.
"""
print('\nTearing down.')
utils.close_driver(self.driver)
self.stop_proxy()
def proxy(self):
"""
Create proxy settings for our Firefox profile.
:return: Proxy
"""
proxy_url = '{0}:{1}'.format(self.proxy_address, self.proxy_port)
p = Proxy({
'proxyType': ProxyType.MANUAL,
'httpProxy': proxy_url,
'ftpProxy': proxy_url,
'sslProxy': proxy_url,
'noProxy': 'localhost, 127.0.0.1'
})
return p
def start_proxy(self):
"""
Start the proxy process.
"""
if not self.proxy_referer:
raise Exception('Set the proxy_referer in child class!')
python_path = sys.executable
current_dir = os.path.dirname(__file__)
proxy_file = os.path.normpath(os.path.join(current_dir, 'referer_proxy.py'))
command = self.proxy_command.format(
python_path, proxy_file, self.proxy_referer, self.proxy_port)
print('Running the proxy command:')
print(command)
self.proxy_server = pexpect.spawnu(command)
self.proxy_server.expect_exact(u'Running...', 2)
def stop_proxy(self):
"""
Override in child class to use a proxy.
"""
print('Stopping proxy server...')
self.proxy_server.close(True)
print('Proxy server stopped.')
I wanted my unit tests to start and stop the proxy server without any user interaction, and could not find any Python samples doing that. Which is why I created the github repo (link above).
Hope this helps someone.
Not sure if i understand your question correctly, but if you want to override your HTTP requests there is no way to do it directly with webdriver. You must run your request thru a proxy. I prefer using browsermob, you can get it thru maven or similar.
ProxyServer server = new ProxyServer(proxy_port); //net.lightbody.bmp.proxy.ProxyServer;
server.start();
server.setCaptureHeaders(true);
Proxy proxy = server.seleniumProxy(); //org.openqa.selenium.Proxy
proxy.setHttpProxy("localhost").setSslProxy("localhost");
server.addRequestInterceptor(new RequestInterceptor() {
#Override
public void process(BrowserMobHttpRequest browserMobHttpRequest, Har har) {
browserMobHttpRequest.addRequestHeader("Referer", "blabla");
}
});
// configure it as a desired capability
DesiredCapabilities capabilities = new DesiredCapabilities();
capabilities.setCapability(CapabilityType.PROXY, proxy);
// start the driver
driver = new FirefoxDriver(capabilities);
Or black/whitelist anything:
server.blacklistRequests("https?://.*\\.google-analytics\\.com/.*", 410);
server.whitelistRequests("https?://*.*.yoursite.com/.*. https://*.*.someOtherYourSite.*".split(","), 200);