Background: Here is my situation, I have a website I access that requires me to go through a proxy. For day-to-day use, I installed the chrome extension Proxy switchyomega to switch between proxies based on a URL wildcard. For example, when I go to URL abc.xyz.com my company requires me to use a proxy at x.x.x.x:yyyy.
Issue I am trying to write an automation script using WebdriverIO (version 7.24.0) in standalone mode using chrome that will launch the website abc.xyz.com. The problem is when I run the script it is not going through the proxy, so my access is blocked. I have tried to configure a proxy in the capabilities but that does not seem to work, and I'm wondering if I'm doing something wrong in my configuration, which is below
const { remote } = require("webdriverio");
const browser = await remote({
capabilities: {
browserName: "chrome",
proxy: {
proxyType: "manual",
httpProxy: "x.x.x.x:yyyy"
}
},
});
await browser.url("abc.xyz.com");
As mentioned in normal use when I access URL abc.xyz.com it routes through the x.x.x.x:yyyy proxy and that is what I need to happen when running my script but the above configuration does not seem to do anything.
Any help would be appreciated.
Related
I am using a puppeteer plugin that opens a webserver when I need to manually solve a captcha. The below code works in development, but I can't figure out an approach to get it to work in production.
I am deploying this app on render.com, and as far as I know, I can only listen to one port.
const app = express();
app.listen(process.env.PORT || "3000");
// I eventually get a link like this
// http://localhost:3001/?targetId=68C3007E851659A5D54CD6E023022C91
puppeteer.use(
PortalPlugin({
// This is a typical configuration when hosting behind a secured reverse proxy
webPortalConfig: {
listenOpts: {
port: 3001,
},
baseUrl: "http://localhost:3001",
},
})
);
I've tried making the port the same on both servers but as expected I kept getting the "port already in use" error. I've also tried pointing the base url to my render.com URL, but it times out when I navigate to the url supplied by the plugin.
The website I am trying to access has ssl certificate-errors
I am using this version of puppeteer "puppeteer": "1.13.0".
When I try to await page.goto('http://bad_ssl_certificate_website') I have timeout error on google cloud only.
TimeoutError: Navigation Timeout Exceeded:
However, It works perfectly fine locally on MAC.
I think the problem is ssl-certificate-errors for my website, because if I try with "google.com" it works okay in both environments.
I used https://www.sslshopper.com to check ssl certificates,and It mentioned this.
The certificate is not trusted in all web browsers. You may need to
install an Intermediate/chain certificate to link it to a trusted root
certificate. Learn more about this error. You can fix this by
following DigiCert's Certificate Installation Instructions for your
server platform. Pay attention to the parts about Intermediate
certificates.
When I was using older version of puppeteer I had problems locally as well.
I saw the exactly the same error
'TimeoutError: Navigation Timeout Exceeded:'
Updating to the newest version of puppeteer has fixed only running the puppeteer locally, but it has not fixed the puppeteer running on google cloud
This is how I setup puppeteer to lunch.
const browser = await puppeteer.launch({
headless: true,
ignoreHTTPSErrors: true,
args: [
"--proxy-server='direct://'",
'--proxy-bypass-list=*',
'--disable-gpu',
'--disable-dev-shm-usage',
'--disable-setuid-sandbox',
'--no-first-run',
'--no-sandbox',
'--no-zygote',
'--single-process',
'--ignore-certificate-errors',
'--ignore-certificate-errors-spki-list',
'--enable-features=NetworkService'
]
});
I found some related issues:
https://bugs.chromium.org/p/chromium/issues/detail?id=877075
Just turn .IgnoreHTTPSErrors = True in the LaunchAsync constructor.
Example:
await new BrowserFetcher().DownloadAsync(BrowserFetcher.DefaultRevision);
var browser = await Puppeteer.LaunchAsync(new LaunchOptions()
{
Headless: true,
IgnoreHTTPSErrors: true
});
The --ignore-certificate-errors-spki-list actually accepts a whitelist of public key hashes ignore certificate-related errors. So it is used like: --ignore-certificate-errors-spki-list=jc7r1tE54FOO=
Chromium doc
I'm having a ton of problems getting an ASP.NET Core 2.1 web application up and running. I need it to run under http.sys (WebListener) on a shared port (80 or 443). I'd also like it to automatically redirect from http (80) to https (443). Of course, I don't want to hard code the listener addresses for http.sys - I need to pull those from a configuration file, but they're hard coded for now. I reserved the appropriate URLs with netsh, but when I run the app I get a warning:
warn: Microsoft.AspNetCore.Server.HttpSys.MessagePump[0]
Overriding address(es) 'http://sharedhost.vbcoa.com:80/app/, https://sharedhost.vbcoa.com:443/app/'. Binding to endpoints added to UrlPrefixes instead.
The app starts, but I can't browse to it with Microsoft Edge at all. Any other web browser is fine - as long as I disable HTTPS. For some reason, the application is forwarding to port 5001, instead of 443.
I figured all of this out. There are four problems. I'll address them each individually.
When configuring http.sys, a warning is issued about overriding local URLs
The UseHttpSys extension method of IWebHostBuilder accepts an options argument with a UrlPrefixes property. However, this is not where you should configure URLs - even if you're using http.sys. You can hardcode them with the UseUrls extension method of IWebHostBuilder, but it would be better to pull it from configuration, which leads to the second problem:
Configuration should be read from appsettings.json
To specify which URLs you want to run the application on, add them to the "urls" element in appsettings.json, as follows:
{
"Logging": {
"LogLevel": {
"Default": "Warning"
}
},
"AllowedHosts": "*",
"urls": "http://sharedhost.vbcoa.com:80/app/;https://sharedhost.vbcoa.com:443/app/"
}
Then you'll need to create a ConfigurationBuilder object, add the appsettings.json file to it, build the configuration (with the Build method) and tell IWebHostBuilder to use that configuration, with the UseConfiguration extension method:
public static void Main(string[] args)
{
var configBuilder = new ConfigurationBuilder()
.SetBasePath(Directory.GetCurrentDirectory())
.AddJsonFile("appsettings.json");
var hostBuilder = WebHost.CreateDefaultBuilder(args)
.UseConfiguration(configBuilder.Build())
.UseHttpSys()
.UseStartup<Startup>();
hostBuilder.Build().Run();
}
Redirection goes to port 5001, not 443
HTTPS redirection is specified in the Configure method of Startup - that functionality comes out of the box. However, by default it will forward to port 5001, even if you have another port specified in your bound URLs from above. To override it, you need to inject HTTPS redirection options via a service. That's handled in the ConfigureServices method of Startup. Add the following line to that method:
services.AddHttpsRedirection(options => { options.HttpsPort = 443; });
Microsoft Edge won't show the web app, even though every other browser will
This is a problem with localhost loopback isolation in Windows Store apps. It seems to affect Windows 10 Enterprise, as discussed here: Microsoft Edge is not able to recognize localhost. To correct it, you need to do two things:
Make sure "Allow localhost loopback" is checked in Edge's "about:flags" page.
Launch a Command Prompt or Powershell Prompt as an Administrator and enter the following:
CheckNetIsolation LoopbackExempt -a -n=Microsoft.MicrosoftEdge_8wekyb3d8bbwe
That should do it!
My Selenium tests are being slowed by third-party scripts that are not necessary to the tests.
How can I block them? Preferably I'd like to block requests to everywhere but localhost.
Solutions offered elsewhere online are:
Block unwanted domains (e.g., *.facebook.com) by editing your hostfiles.
Route all your tests through BrowserMob which can be configured to filter requests.
Both options seemed like overkill to me. Editing the host files affects your whole system, and using BrowserMob introduces new problems.
Here's another way: Use a PAC file to configure the browser to connect to localhost directly, and attempt to connect to everything else through an unavailable proxy.
Selenium code (Java):
DesiredCapabilities capabilities = DesiredCapabilities.chrome();
Proxy proxy = new Proxy();
proxy.setProxyType(Proxy.ProxyType.PAC);
proxy.setProxyAutoconfigUrl("http://localhost:8080/my-pac-file.pac");
capabilities.setCapability("proxy", proxy);
ChromeDriver driver = new ChromeDriver(INSTANCE, capabilities);
The PAC file:
function FindProxyForURL(url, host) {
if (host.toLowerCase() === "localhost"){
return "DIRECT"; // whitelisted
}
return "PROXY 127.0.0.1:9876"; // blocked (bad proxy)
}
I've been having an issue for days and I don't know how to fix it.
I am trying to setup my SSL certificate, and for some reason the site works on http, and then when I try to load https, it loads only the navbar and sidebar, and then it's stuck on the spinner.
When I examine at the network connections on chrome, it keeps trying to load xhr and websockets.
In safari I get this error in the console
WebSocket connection to 'wss://mysite/sockjs/530/72iokiqa/websocket' failed: WebSocket is closed before the connection is established
I am trying to set the headers, in particular the x-forwarded-proto header, but I can't figure out how to do that.
I am using mup.
// Configure environment
"env": {
"ROOT_URL": "https://inslim.com"
},
"ssl": {
"pem": "./ssl.pem"
}
For some reason, when I try to add a por to the env variable, it won't allow me to do mup deploy. It will break and the site will go down.
I am also confused with nginx. I installed it and I set it up, but I don't think it's making any difference. If I run 'service nginx stop' or service nginx start, it doesn't make any difference.
Can someone help me? Any advice or anything would help. Or if you need any other info please let me know.
Here's a screenshot of my spinner of death
The ssl part of your configuration JSON looks fine, but your env part needs a little modification. The env part of the configuration JSON should at least look something like this:
"env": {
"PORT": 80, // Defaults to 80, but could be different if app is configured differently
"ROOT_URL": "http://inslim.com"
}
If you do not have the force-ssl package already added to your application, I would suggest adding that (don't worry, it is a core Meteor package). If you do not also have the spiderable package added to your application, then your ROOT_URL element in your JSON can remain prefixed with http, but if you do have the spiderable package added to your application, you will need to change that ROOT_URL element prefix in your JSON to be https. All of this information is per the documentation for Meteor Up, which can be found here. Also, I can confirm that this setup with the JSON works because I have a production application that is running with this exact setup without any issues.