(Resolved, see below)
I am testing an app that uses an external IdP for user login with Webdriver IO to control Selenium.
I need to use IndexedDB to save state (login state, tokens).
Problem is, when testing with Selenium, indexedDB is cleared when navigating outside the domain to the IdP. And on return to the app, the necessary state is lost. localStore (which is not an option for my app) also fails to persist when tested.
1, start at www.example-app.com, save state to indexedDB or localStorage
2, go to www.example-idp.com
3, return to www.example-app.com indexedDB and localStorage are completely empty!!
Stack:
Webdriver IO with Jasmine
Selenium Standalone, Chrome Driver (configurations for both)
React SPA
UPDATE: Issue is resolved!
So embarrassed to admit but the redirect URL was to a different port (and a different hosted copy of the app) which explains why the all local storage mediums that are origin-locked were empty. I didn't even realize till now... 😏
If you have a similar problem, please double check to make sure your domain and ports match. 👍
Related
I am trying to automate login to my app which uses among others, google sso authentication.
However login form return error "This browser or app may not be secure.". I set my google account options to allow less secure apps but still nothing.
I browsed few topics:
GMail is blocking login via Automation (Selenium)
Selenium Google Login Block
Automation Google login with python and selenium shows ""This browser or app may be not secure""
And it seems that google is blocking this way at all in favor of oauth.
People write in these topics that solutions stopped working recently
So is it currently possible, to set ChromeDriver somehow using capabalities, to be able to login through SSO?. I need a simple solution, that will run headless with other scripts on cloud (not something that would require me to manually login first on another instance as one anwser suggests).
If its not possible or extremly complicated please tell me I will not waste time on it.
If you want to use chrome capabilities, what you can do is set the user-data-dir to a chrome profile that has already been signed in using SSO.
You should look up how to reuse chrome profiles with selenium.
If your accounts have 2 steps verifications, google believe it's safer and allows you to get login. Then the issue will be how to handle the 2 steps verifications. Working on that :/
I'm curious how reCAPTCHA v3 works. Specifically the browser fingerprinting.
When I launch an instance of Chrome through Selenium/chromedriver and test against reCAPTCHA 3 (https://recaptcha-demo.appspot.com/recaptcha-v3-request-scores.php) I always get a score of 0.1 when using Selenium/chromedriver.
When using incognito with a normal instance, I get 0.3.
I've beaten other detection systems by injecting JavaScript and modifying the web driver object and recompiling webdriver from source and modifying the $cdc_ variables.
I can see what looks like some obfuscated POST back to the server, so I'm going to start digging there.
What might it be looking for to determine if I'm running Selenium/chromedriver?
reCaptcha
Websites can easily detect the network traffic and identify your program as a BOT. Google have already released 5(five) reCAPTCHA to choose from when creating a new site. While four of them are active and reCAPTCHA v1 being shutdown.
reCAPTCHA versions and types
reCAPTCHA v3 (verify requests with a score): reCAPTCHA v3 allows you to verify if an interaction is legitimate without any user interaction. It is a pure JavaScript API returning a score, giving you the ability to take action in the context of your site: for instance requiring additional factors of authentication, sending a post to moderation, or throttling bots that may be scraping content.
reCAPTCHA v2 - "I'm not a robot" Checkbox: The "I'm not a robot" Checkbox requires the user to click a checkbox indicating the user is not a robot. This will either pass the user immediately (with No CAPTCHA) or challenge them to validate whether or not they are human. This is the simplest option to integrate with and only requires two lines of HTML to render the checkbox.
reCAPTCHA v2 - Invisible reCAPTCHA badge: The invisible reCAPTCHA badge does not require the user to click on a checkbox, instead it is invoked directly when the user clicks on an existing button on your site or can be invoked via a JavaScript API call. The integration requires a JavaScript callback when reCAPTCHA verification is complete. By default only the most suspicious traffic will be prompted to solve a captcha. To alter this behavior edit your site security preference under advanced settings.
reCAPTCHA v2 - Android: The reCAPTCHA Android library is part of the Google Play services SafetyNet APIs. This library provides native Android APIs that you can integrate directly into an app. You should set up Google Play services in your app and connect to the GoogleApiClient before invoking the reCAPTCHA API. This will either pass the user through immediately (without a CAPTCHA prompt) or challenge them to validate whether they are human.
reCAPTCHA v1: reCAPTCHA v1 has been shut down since March 2018.
Solution
However there are some generic approaches to avoid getting detected while web-scraping:
The first and foremost attribute a website can determine your script/program is through your monitor size. So it is recommended not to use the conventional Viewport.
If you need to send multiple requests to a website keep on changing the User Agent on each request. Here you can find a detailed discussion on Way to change Google Chrome user agent in Selenium?
To simulate human like behavior you may require to slow down the script execution even beyond WebDriverWait and expected_conditions inducing time.sleep(secs). Here you can find a detailed discussion on How to sleep webdriver in python for milliseconds
Outro
Some food for thought:
Selenium webdriver: Modifying navigator.webdriver flag to prevent selenium detection
Unable to use Selenium to automate Chase site login
Confidence Score of the request using reCAPTCHA v3 API
Selenium and Puppeteer have some browser configurations that is different from a non-automated browser. Also, since some JavaScript functions are injected into browser to manipulate elements, you need to create some override to avoid detections.
There are some good articles explaining some points about Selenium and Puppeteer detection while it runs on a site with detection mechanisms:
Detecting Chrome headless, new techniques - You can use it to write defensive code for your bot.
It is not possible to detect and block Google Chrome headless - it explains in a clear and sound way the differences that JavaScript code can detect between a browser launched by automated software and a real one, and also how to fake it.
GitHub - headless-cat-n-mouse - Example using Puppeteer + Python to avoid detection
I want to automate an application for an upload process with selenium.Therefore I am using geckodriver. Right now I am doing the login site headless but I want to do the login by the user and then change to headless. Is there a way to do that or at least a work around?
There is no way to do that using the same browser instance... The headless flag is a setting passed to the browser on startup and there is no way to dynamically change that.
If the site uses cookie-based authentication, here is an alternate approach:
login with browser in normal (GUI) mode
export saved cookies
instantiate a new browser in headless mode
navigate to the site again with the headless driver
add saved cookies to the new headless driver
... at that point, you can navigate to an authenticated page. You should be "logged in" since you are re-using the cookies from the original browser session.
I'm creating a app in Ionic 2, wich consumes a web api from an existing site. To use this API i have to make athenticate in it in the following way (Similar to facebook login):
I call the api login page in a InAppBrowser component, sending the proper keys and a return URL.
the user types the login and password in the form displayed, the API will validate it and authenticate it.
The API calls the return URL passing the authorization token.
I 'hijack' this redirect to the return url in the InAppBrowser 'loadstart' event, and extract and store the authorization token.
In the following calls to the API, i send the authorization token in the header.
This is all working fine in the emulator, but it doesn't work in the browser (with ionic serve), because when i call InAppBrowser it actually calls window.open, and the events doesnt work. I can't detect the redirect action made in the opened window.
I'd like to make this work in the browser since its better to debug the application there. My first thought was to send "http://localhost:8001" as the return url, but I couldn't find a way to catch the token parameter in the ionic application.
Does anyone know how I can catch this parameter or any other way to make this login work in the browser? It is for development and debug purposes only, so strict security is not a issue (I can comment out any unsecure code in the production version).
Edit: Hayden Braxton answer didn't solve my problem, but since it was because of something exclusively to my app, and it could really help someone who wants to make plugins work, I'll keep it as the selected answer.
Besides that, I'll share the solution I found to my problem in case it could help anyone. It was simple, actually:
I pass "http://localhost:8001" as the api return_uri parameter
the api will, after checking the login and password, redirect to http://localhost:8001?token=MY_AUTH_TOKEN.
This will reload the application and call login page again.
In the login page i call this.platform.getQueryParam("token"); to get the token.
Add
"browser": "ionic-app-scripts serve --iscordovaserve --sourceMap source-map --wwwDir platforms/browser/www/ --buildDir platforms/browser/www/build",
to the script section of your package.json. Then instead of doing ionic serve, instead run
npm run browser
We use ionic2 to develop our apps where I work, and this is what we figured out after some research.
Before using this, you need to have the browser platform added. You can accomplish this with the following:
ionic add platform browser
If the browser platform is already added, delete the browser directory from your platforms directory and then run the add platform command, just to be on the safe side.
In my web application, I have used the asp Login control to facilitate the login process. I have noticed that if I open the site in a browser and log in, and then open another browser and go to the site, then the second browser shows that I am already logged in, even though I didn't log in using that browser. I have noticed that some other sites (such as my bank's web access) work this way, but it still seems odd to me. I'm not sure if it's bad or not, but it is strange.
Is this behavior bad?
I have also noticed that if I close all of my browsers and then open a new one up and go to the site, I am NOT logged in.
This is because those sites use session cookies. If you open different brwosers at the same time, such as FireFox, Opera, Chrome and IE, you'll have to log in in all of the browsers independently. However, when re-using the same browser application, this usually does not spawn a new process with its own session data but rather re-uses the already open browser.
If you are using a browser that supports multi-tabbed sessions e.g. IE7+, Firefox etc, you will find that, if you have the same ASP.NET website open in multiple tabs, each tab will share the same authentication credentials. This is because the session ID applies to the browser instance, not the tab instance, so if a user logs in on 1 tab, then opens up another tab to the website, they won't have to provide their credentials twice.
I don't think there's an easy way around this behaviour. You could choose employ cookie 'munging', storing the forms authentication ticket on the URL, but this seems like a bad idea from a security POV.
Lucero and pmarflee are both correct.
To provide a bit more information though:
Firefox will share session between tabs and all browser instances
IE 8 will also share session between tabs and all browser instances
IE 7 shares session between tabs, but not across instances
IE 6 does not share session across instances
I'm not sure how Opera, Chrome, or Safari handle sessions.