BrowserMob Proxy and Webdriverio minimal, empty HAR file - selenium

I was wondering if anyone has any idea as to what could be the problem or additional steps I could take in resolving the following issue.
I have a requirement to capture the network traffic so I can write some selenium tests verifying certain request headers.
The problem is when trying to use Webdriver.io + Browsermob proxy the HAR file created contains very minimal information.
I have tried using the C# bindings of the proxy and that resulted in the same issue as the Javascript ones. The only bindings that I got to work which returned data in the HAR file were the Java ones. There's a business requirement though that I must use Webdriver.io to implement this.
I have checked a number of other Questions and Answers from stack overflow but to no avail. Most have not been answered and the ones that have been have not worked for me just yet.
The code I am currently running is very similar to this one with the difference being the URL I'm trying to get the information from.
The difference between my Java code which worked and the Javascript and C# ones was that the Java one sets up the proxy programmatically whereas the other two expect the proxy to be already running and just connect to it.
Even with that in mind, when I start the C# or Javascript tests the proxy registers the new connection. The selenium server also starts up fine. Both cleanup with no issues according to their respective consoles once the tests are finished.
Are there any other ways to potentially debug this? Or even alternatives to capture the network traffic (Must work cross browser - already have a solution which works for chrome using chrome-remote-interface but we saw suggestions to use browsermob proxy for cross browser network capture).
Thanks for your time
Something I forgot to mention which is pretty important. Below is the npm package I am using for the browsermob-proxy :
https://www.npmjs.com/package/browsermob-proxy

Okay so, I figured out why I wasn't getting any data. I hope this saves others some time and hassle.
The problem is the way browsermob proxy handles localhost. I instead switched the proxy to use my IPV4 address and it started capturing all the HAR data.
See the code below:
//BroswerMobProxy + webdriver.io
//npm package used https://www.npmjs.com/package/browsermob-proxy
var webdriverio = require('webdriverio');
//proxy settings, host is IPV4 address
var Proxy = require('browsermob-proxy').Proxy
, fs = require('fs')
, proxy = new Proxy({host: 'Put IPV4 Address Here', proxyPort: 8081 , selHost: 'Put IPV4 Address Here'});
;
//convenience method that
proxy.cbHAR({captureHeaders: true, captureContent: true, captureBinaryContent: true }, doSeleniumStuff, function(err, data) {
if (err) {
console.error('ERR: ' + err);
} else {
/* Make sure har results are in the correct shape
* for any further processing
*/
var harResultsString = JSON.stringify(data);
var harResultsJson = JSON.parse(harResultsString);
//Write HAR file
fs.writeFileSync('DemoFile.json', harResultsJson, 'utf8');
//Print to console
console.log(harResultsJson);
}
});
//webdriver.io options
const opts = {
desiredCapabilities:{
browserName: 'chrome',
proxy: {
proxyType: 'MANUAL',
httpProxy: String(proxy.host)+":"+String(proxy.proxyPort),
sslProxy: String(proxy.host)+":"+String(proxy.proxyPort),
socksProxy:String(proxy.host)+":"+String(proxy.proxyPort),
socksVersion: 4,
autodetect: false
},
acceptSslCerts: true,
acceptInsecureCerts: true
},
host: 'Put IPV4 Address here',
port: 4444,
protocol: 'http',
coloredLogs: true,
proxy: 'http://'+String(proxy.host)+":"+String(proxy.proxyPort),
}
function doSeleniumStuff(proxy, cb) {
var browser = webdriverio.remote(opts);
// console.log(browser.options);
browser
.init()
.url('http://yahoo.com.au')
.getTitle().then(function(title) {
console.log('Title was: ' + title);
})
.end().then(cb).catch(e => console.log(e));
}

Related

chrome:headless (MacOS) results with ' 1) AssertionError: expected 'about:blank' to include $target page'

I am using TestCafe in combination with gherkinTestcafe (steps) / cucumber.
I am also using environment variables so that i can run my tests on 2 different environments.
My code is as follows, although through debugging, i don't believe this is something strictly code related, as much as it is related to:
chrome:headless
environment
version of chrome / MacOS
import Enviorments from "../../../../../../AEM_Engine/Enviorment/Enviorments";
import { Helper } from "../../../../../TestActions/Test_specific/Career_helper";
import {AddAuthCredentialsHook} from "../../../../../TestActions/BasicAuth";
const {Before, Given, Then} = require('cucumber');
let publisher = new Publish();
let aemEnv = new Enviorments();
let helper = new Helper;
let careersPage = '/career';
Before('#basicAuth', async testController => {
const addAuthCredentialsHook = new AddAuthCredentialsHook('$someUserName', '$somePassword');
await testController.addRequestHooks(addAuthCredentialsHook);
});
Before('#disableCookie', async testController => {
await testController.addRequestHooks(publisher.mockCookieResponse);
});
Given('I am at Careers page', async testController => {
await publisher.Navigate(testController, aemEnv.frontEndURL + careersPage);
await publisher.verifyURL(testController, aemEnv.frontEndURL + careersPage);
});
.
.
.
When i wait for the script to run i have
1) AssertionError: expected 'about:blank' to include $expectedPage
As i mentioned, i don't believe the problem is in the code. Even if i remove the step for verifying the current URL location, the test fails on the next step after.
Tests pass on
Chrome (with UI shell)
Other browsers (firefox, safari), headless or with UI shell
Second (staging) environment
When Tests are run and TestCafe starts, i get the following info
Running tests in:
- HeadlessChrome 99.0.4844 / Mac OS X 10.15.7
Feature: Careers Page Available
(node:87344) Warning: Setting the NODE_TLS_REJECT_UNAUTHORIZED environment variable to '0' makes TLS connections and HTTPS requests insecure by disabling certificate verification.
I tried re-installing some packages, re-writing some of the steps, adding some flags to clear cache, change chrome port or similar, but nothing worked.
Any thoughts on what might be causing this and how to solve it?

Not able to close the browser using NightwatchJS

I am trying to open my url using Nightwatch and I wan't able to close the browser afterwards.
I tried using timeouts, as well as browser.end(), or browser.closeWindow(). None of them seem to be working for my url.
module.exports = {
'Demo test mywrkouts' : function (browser) {
browser.url('https://www.mywrkouts.com/workouts/search')
browser.timeouts('script', 10000, function(result) {
browser.end();
console.log("Test result"+result);
});
//browser.closeWindow();
}
};
It opens the page, but doesn't close the browser. I am using Chrome browser with chromedriver. I am expecting to close the window, but it doesn't work.
Any advice is appreciated.
LE: Like I extensibly described below, you don't need to explicitly close the browser at the end of the test (via browser.end()) as the Nightwatch test-runner does that for you at the end of each feature-file.
But, if you need to do some teardown operations and then explicitly close the session, do it in an after (or afterEach) hook. Try the following snippet:
module.exports = {
before(browser) {
browser.maximizeWindow();
},
'My Wrkouts Test': (browser) => {
browser.url('https://www.mywrkouts.com/');
// Check if the website logo is visible:
browser.expect.element('#barbell-homepage-top-image-desktop img.app-bar-desktop-logo').to.be.visible;
// Check the articles heading text:
browser.expect.element('h3.blog-carousel-title.primary-blue-text.center').text.to.contain('Foundational Education Series');
},
after(browser, done) {
browser.end(() => {
console.info('*--*--*--*--*--*--*--*--*--*--*--*--*');
console.info('*-- Clossing session... Good bye! --*');
console.info('*--*--*--*--*--*--*--*--*--*--*--*--*');
done();
});
}
};
Anyways, I feel you are confusing the way NightwatchJS/WebdriverIO/Protractor (or any other Webdriver-based test solution) is handling a browser session.
First off, you need not worry about closing the active session. Nightwatch does it for you at the end of each test feature-file. Thus, running a suit of let's say three test suites (login.js, register.js, forgot_password.js) will sequentially spawn & close three different browser sessions.
Also, browser.closeWindow() is only used for closing a window instance (taking into account that you have multiple windows associated with the same browser session). It won't close your main window, unless you have switched to another window instance (which was previously opened during your test run).
If you use browser.end() in the middle of your test, then you basically kill the active session, nullifying the following logic from your feature-file:
INFO Request: DELETE /wd/hub/session/4a4bb4cb1b38409ee466b0fc8af78101
- data:
- headers: {"Content-Length":0,"Authorization":"Basic Z29wcm86YmM3MDk2MGYtZGE0Yy00OGUyLTk5MGMtMzA5MmNmZGJhZTMz"}
INFO Response 200 DELETE /wd/hub/session/4a4bb4cb1b38409ee466b0fc8af78101 (56ms) { sessionId: '4a4bb4cb1b38409ee466b0fc8af78101',
status: 0,
value: null }
LOG → Completed command end (57 ms)
Everything after will look like this:
INFO Response 404 POST /wd/hub/session/null/elements (11ms) { sessionId: 'null',
value:
{ error: 'invalid session id',
message: 'No active session with ID null',
stacktrace: '' },
status: 6 }
!Note: There is no support for doing what you are trying to do, nor is it a common use-case, thus the lack of support for it across
all of these testing solutions.
They say a picture is worth 1000 words, so let's me simply put it this way... what you are trying to do is synonymous with the following:

How to serve data for AJAX calls in a Vue.js-CLI project?

I have a Vue.js CLI project working.
It accesses data via AJAX from localhost port 8080 served by Apache.
After I build the project and copy it to a folder served by Apache, it works fine and can access data via AJAX on that server.
However, during development, since the Vue.js CLI website is being served by Node.js which is serving on a different port (8081), I get a cross-site scripting error) and want to avoid cross-site scripting in general.
What is a way that I could emulate the data being provided, e.g. some kind of server script within the Vue.js-CLI project that would serve mock data on port 8081 for the AJAX calls during the development process, and thus avoid all cross-site scripting issues?
Addendum
In my config/index.js file, I added a proxyTable:
dev: {
env: require("./dev.env"),
port: 8081,
autoOpenBrowser: true,
assetsSubDirectory: "static",
assetsPublicPath: "/",
proxyTable: {
"/api": {
target: "http://localhost/data.php",
changeOrigin: true
}
},
And now I make my AJAX call like this:
axios({
method: 'post',
url: '/api',
data: {
smartTaskIdCode: 'activityReport',
yearMonth: '2017-09',
pathRewrite: {
"^/api": ""
}
}
But now I see in my JavaScript console:
Error: Request failed with status code 404
Addendum 2
Apparent axios has a problem with rerouting, so I tried it with vue-resource but this code is showing an error:
var data = {
smartTaskIdCode: 'pageActivityByMonth',
yearMonth: '2017-09'
}
this.$http.post('/api', data).then(response => {
this.pageStatus = 'displaying';
this.activity = response.data['activity'];
console.log(this.activity);
}, response => {
this.pageStatus = 'displaying';
console.log('there was an error');
});
The webpack template has its own documentation, and it has a chapter about API proxying during development:
http://vuejs-templates.github.io/webpack/proxy.html
If you use that, it means that you will request your data from the node server during development (and the node server will proxy< the request to your real backend), and the real backend directly in production, so you will have to use different hostnames in each environment.
For that, you can define an env variable in /config/dev.env.js & /config.prod.env.js

WebRTC TURN Server - I set it up, but it's not working

I've set up a text chat service using the PeerJS implementation of WebRTC's data channel. PeerJS provides a basic signalling server for this purpose, but I have tried to replace that with STUN and TURN servers set up through XirSys (recommended by SimpleWebRTC, another WebRTC library). I haven't deployed to the web yet.
Using Node to serve my static files locally, it will work on a local network (when I am sitting next to the person and they navigate to my ip/port in the browser), but will not work when connecting through different access points on the same network (i.e. at work, on opposite ends of the building).
My hypothesis is that it's hitting a firewall, but still directing traffic to PeerJS' signalling server without falling back to the XirSys STUN and TURN servers I've tried to set up. Here's the code I'm working with:
var stun = {};
var turn1 = {};
var turn2 = {};
$.ajax({
type: "POST",
dataType: "json",
url: "https://api.xirsys.com/getIceServers",
data: {
ident: "myusername",
secret: "long-alphanumeric-secret-key",
domain: "www.adomain.com",
application: "anapp",
room: "aroom",
secure: 1
},
success: function (data, status) {
console.log(data);
stun = data.d.iceServers[0];
turn1 = data.d.iceServers[1];
turn2 = data.d.iceServers[2];
},
async: false
});
var conn;
// Connect to PeerJS, have server assign an ID instead of providing one
var peerID = prompt('What would you like your screen name to be?');
var peer = new Peer(
peerID,
{key: 'mypeerjsserverkey', debug: true},
{
config: {'iceServers': [
{url: stun.url},
{url: turn1.url, credential: turn1.credential, username: turn1.username},
{url: turn2.url, credential: turn2.credential, username: turn2.username}
]
}
});
NOTE: My ident, secret, domain, etc. obviously aren't accurately represented here. I don't think that's where my problem is.
Any thoughts?
If you email us a wireshark capture of your STUN/TURN traffic, we should be able to outline where your problem is. Messages sent over signalling are separate but parallel to WebRTC messages. Therefore, if the app is working but the messages are being sent over signalling, then it's possible the configuration of the application isn't correct.
XirSys provides TURN via UDP over TCP through port 80/443, so if the signalling is connecting and flowing, so should the TURN.
Also, looking at your code, if you pass data.d from your getIceServers success handler to the PeerJS config, that should reduce your code quite a bit :-) The ICE string you're reconstructing doesn't need to be broken down.
Regards,
Lee
XirSys CTO

Socket Hang Up when using https.request in node.js

When using https.request with node.js v04.7, I get the following error:
Error: socket hang up
at CleartextStream.<anonymous> (http.js:1272:45)
at CleartextStream.emit (events.js:61:17)
at Array.<anonymous> (tls.js:617:22)
at EventEmitter._tickCallback (node.js:126:26)
Simplified code that will generate the error:
var https = require('https')
, fs = require('fs')
var options = {
host: 'localhost'
, port: 8000
, key: fs.readFileSync('../../test-key.pem')
, cert: fs.readFileSync('../../test-cert.pem')
}
// Set up server and start listening
https.createServer(function (req, res) {
res.writeHead(200, {'Content-Type': 'text/plain'})
res.end('success')
}).listen(options.port, options.host)
// Wait a second to let the server start up
setTimeout(function() {
var clientRequest = https.request(options, function(res) {
res.on('data', function (chunk) {
console.log('Called')
})
})
clientRequest.write('')
clientRequest.end()
}, 1000)
I get the error even with the server and client running on different node instances and have tested with port 8000, 3000, and 443 and with and without the SSL certificates. I do have libssl and libssl-dev on my Ubuntu machine.
Any ideas on what could be the cause?
In
https.createServer(function (req, res) {
you are missing options when you create the server, should be:
https.createServer(options, function (req, res) {
with your key and cert inside
I had a very similar problem where the response's end event never fired.
Adding this line fixed the problem:
// Hack to emit end on close because of a core bug that never fires end
response.on('close', function () {response.emit('end')});
I found an example of this in the request library mentioned in the previous answer.
Short answer: Use the the latest source code instead of the one you have. Store it where you will and then require it, you are good to go.
In the request 1.2.0 source code, main.js line 76, I see
http.createClient(options.uri.port, options.uri.hostname, options.uri.protocol === 'https:');
Looking at the http.js source code, I see
exports.createClient = function(port, host) {
var c = new Client();
c.port = port;
c.host = host;
return c;
};
It is requesting with 3 params but the actual function only has 2. The functionality is replaced with a separate module for https.
Looking at the latest main.js source code, I see dramatic changes. The most important is the addition of require('https').
It appears that request has been fixed but never re-released. Fortunately, the fix seems to work if you just copy manually from the raw view of the latest main.js source code and use it instead.
I had a similar problem and i think i got a fix. but then I have another socket problem.
See my solution here: http://groups.google.com/group/nodejs/browse_thread/thread/9189df2597aa199e/b83b16c08a051706?lnk=gst&q=hang+up#b83b16c08a051706
key point: use 0.4.8, http.request instead of http.createClient.
However, the new problem is, if I let the program running for long time, (I actually left the program running but no activity during weekend), then I will get socket hang up error when I send a request to http Server. (not even reach the http.request). I don't know if it is because of my code, or it is different problem with http Server