I am writing an puupeteer project for scraping web site.
I use stealth plugin for puppeteer because website detect me as a bot.
And everything was going well until I decided to check my browser on iphey.com
It shows me that I am trying to spoof my hardware
https://prnt.sc/1qk7t01
In details of hardware section I cannot get some useful info except to https://prnt.sc/1qk7wu3
As I understand it is hashs or something like this.
I decided to go further and check my puppeteer instance on https://browserleaks.com/
all the sections of the site are next
ip - https://drive.google.com/file/d/1__U5UK0D2xZVttXPhzUKzn06iUhUR9Jy/view?usp=sharing
rest in the comments bellow
And if I trully understand main problem with my puppeteer is WebGL Vendor and Rernder.
I would like to override them but have no idea how to do that.
Can you help me to ecome undetectable for iphey.com with my instance of puppeteer.
github - https://github.com/Golofaevap/myBrowser
I hope you will help me to find some solution
Related
i am trying to automate tests for 'lens' electron based desktop application.i was following this link enter link description here to setup the test for electron app.This link expects a 'mainwindowurl' but application doesn't have any main page, but testcafe give suggestion of the mainwindowurl as an error so tried it works but am not convinced with the suggrstion urls ,but same way want to give fixture page url on the test what should be the url should i need to give? then have got one more error ERROR Unable to establish one or more of the specified browser connections. This can be caused by network issues or remote device failure.Please can you guys suggest what to do?
Each and every Electron application has to navigate to a page after opening a window.
I guess that the mentioned Lens app is an Kubernetes IDE: https://github.com/lensapp/lens.
This app uses the BrowserWindow.loadURL function to navigate to the main page:
https://github.com/lensapp/lens/blob/a61425124f18b1cc2d8a507084a472029acc3e6b/src/main/window-manager.ts#L101.
Digging the code a bit more, I found that the main window URL is just localhost with some port:
https://github.com/lensapp/lens/blob/a61425124f18b1cc2d8a507084a472029acc3e6b/src/main/window-manager.ts#L33.
I guess it is possible to determine or set the port number by looking at the code a bit more or asking Lens developers about it.
I am aware of the Pyppeteer library and Pyppeteer Stealth, but the problem with them is that the website that I am trying to scrape information from detects Pyppeteer Stealth (Python transplant of Puppetteer) and blocks it. The original Puppetteer Stealth used on node JS does work fine on that website, however, I would much rather create this scraper on Python since I am much more familiar with it.
Which other stealthy and up to date headless browser automation tools are available?
All I will need it for is grabbing the HTML content and parsing it through Beautiful Soup. Unfortunately, the requests and requests-html library also do not work on this website.
If you don't care about the automation part of the software that much I would just recommend looking into Scrapy (and Scrapy Splash if you need js to be rendered which is why I assume you want to use Pyppeteer in the first place) combined with the use of some basic tactics to not get caught as a bot such as user-agent rotation and proxy rotation.
This is the tactic I am using too to make a scraper for similarweb.com at the moment.
I have an Angular(2) + TypeScript application using Keycloak as the authentication/authorization provider,
would like to integrate Keycloak with NativeScript to make the existing web application cater to iOS/Android users.
could see Keycloak supports iOS/Android as listed here : http://www.keycloak.org/docs/2.5/securing_apps_guide/topics/overview/supported-platforms.html?408DD571-0042-4EBB-BC37-1F2966A56651=123451
Is there any NativeScript based plugin to integrate with Keycloak (could not find anything here : http://plugins.nativescript.org/ ) ? pointers for a possible solution would be of great help
Version:
tns --version
2.5.0
Looks like the answer is no. But I start playing around, and to create a plugin, we have basically two strategies:
Extend a pure OAuth client for nativescript. There is this plugin: http://plugins.nativescript.org/plugin/nativescript-oauth
Port the keycloak.js to Nativescript.
I like the second approach because there are more features build in keycloak, so my first try was edit the original code and remove the use of window and other API's that are not present in the browser. I quickly find out that keycloak.js code is a mess (at least to me), and I'm getting a bad time undestanding how to edit its 1258 lines. Because of that, I start a complete rewrite, highly inspired in the original. I'm actualy reading the code and trying to port the project piece by piece, reorganizing for test, and abstracting the use of browser API's to port for web and nativescript.
You can check the code here: https://github.com/atende/keycloak-js-universal but is not close to ready.
The problem is, I have no time to do it :-), but will get done eventually, because I need that too.
Is important to mention that I create a abstraction for Angular 2 that works using the keycloak.js for security, and porting the keycloak.js is strategic to make the library work for Angular 2 Nativescript, which is on my motivation to do it. You can check the project here: https://github.com/atende/angular-spa
Any contributions are wellcome ;-)
When a user downloads a plugin firefox (for example) the plugin installation begins as soon as the download has completed.
Is it possible to achieve the same thing in safari? i.e. user clicks link to download plugin, once it has downloaded it automatically begins the installation.
I don't think this is possible to do on any other domain except extensions.apple.com.
I've done some extensive testing on this and the safari.installExtension() method is only present if the domain matches extensions.apple.com (probably controlled by the browser, similar to how certain Chrome APIs only shows up inside of extensions themselves).
I tested this theory by going to the JS file itself and opening the JS console:
https://extensions.apple.com/home/scripts/extensionInstall.js
After that JS file has loaded, type typeof(safari.installExtension) in the JS console and it should return "function". Notice that it exists on a non-HTML page, meaning it's being provided by the browser (since this script doesn't execute, nor has the code in it to provide this method).
I tried doing this on other sites and it doesn't exist: "undefined".
I also had a crazy thought that it just needs extensions as the subdomain, so I tested it on http://extensions.joomla.org too, no dice. I can't seem to find an extensions sub-domain that's SSL though. That may work, but I seriously doubt it as the method appears to be regulated to only show up when on Apple's specific extensions sub-domain.
No solution for that here, but maybe this can help?
In apple extension gallery at https://extensions.apple.com the extensions do install in one click, and i wanted to achieve the same in a website of mine, so i went and checked their JS source code.
A javascript file there defines a "ExtensionOneClick" Object (https://extensions.apple.com/home/scripts/extensions.js).
A method is dedicated to installing extensions:
// href : path to the extension ".safariextz" file
// id : com.whatever.myextension-<safaridevelopper10charsid>
safari.installExtension(id, href);
I tried to replicate this on my website, but i get "safari is undefined", so I guess a site specific hack in Safari is helping here?
I'm not really sure how to word this exactly, so hopefully someone can make sense of it. I've been working on an iPad app that syncs files from a server to your iPad and lets you build presentations with the various files. The corporation I'm working with on this app has a wireless network that requires you to re-authenticate every hour. So every hour instead of getting the expected JSON api response, any HTTP request pulls down the page needed to reauth with the wireless network. I was wondering is there is a specific HTTP response code related to getting sent that page or a "best-practice" way of testing for that page as opposed to JSON.
Granted I could just test to see if the response is HTML, but that doesn't account for other redirect responses that I haven't found yet. I could just test part of the HTML to see if it matches a predetermined portion of the html, but I'm an outside contractor. I can't guarantee they won't change the markup or verbiage of the page after I've made my deliverable.
So does anyone out in the ether know a "best practices" methodology for testing if the app needs to reauth before syncing?
I noticed that on Mac OS X and maybe even iOS, when you connect to a new Wi-Fi network, it will try to contact www.apple.com. This is being done to check if the internet connectivity is available. If it's not available, the Captive Network Assistant will pop-up, showing you the authentication page, or sometimes when I'm in Starbucks, an advertisement.
Following your question, since Apple themselves is doing it this way, I think you could check for HTTP response code, look for something in the HTML markup (slightly discouraged though) or trying to connect to a known server (Reachability).
For a corporation practicing well-documented projects, I am quite sure they won't be changing things without making sure that your app, once deployed will continue to work.