I am trying to scrape a website, but it is not loading in selenium. When I browse that website in my "real" chrome browser, everything works fine. Is there any way I can use my real browser with python to automate stuff, instead of using selenium??
Thanks
Using selenium we can automate real browsers.
If in case the website is not loading via selenium, you can check if adding desired capabilities helps.
Here we can set proxy, disable extensions etc. There are many options available.
https://chromedriver.chromium.org/capabilities
Also if you can share what kind of error is displayed that would be helpful.
Related
With Robot Framework and its Selenium Library, I need to open specifically Firefox, get to a repository on GitLab and download a certain file. Please don't question the tool choice, I was asked to do this with Robot on Firefox and I have to do it with Robot on Firefox. Nothing crazy on the surface actually, but I found out that GitLab runs a "check" on the browser and apparently Selenium gets stuck.
I've searched for solutions, but they all apply to either Selenium on Java, Python, etc., and most of them are about Chrome. Only a handful of unclear ones talk about Firefox and none about Robot with Selenium. I've tried to adapt some of them, such as the following:
SeleniumLibrary.Open Browser browser=firefox
... ff_profile_dir=set_preference("dom.webdriver.enabled","false");set_preference("useAutomationExtension","false")
SeleniumLibrary.Go To ${UrlGitLab}
But it doesn't work. It's still stuck on the page
Checking your browser before accessing gitlab.com.
This process is automatic. Your browser will redirect to your
requested content shortly.
Please allow up to 5 seconds…
It looks like it tries to reload (?) a couple of times, but it won't go further.
Is there a solution, or a workaround that doesn't completely ditch Robot + Selenium with Firefox?
I am building a web site in Django that would scrape data from some site, so people could enter the site, set custom data filters and view scraped data in friendly format.
The problem is that requests and beautiful soup modules will not be enough for the scraping purposes, since I will also need some automation to be done (loading javascript or clicking buttons).
Since Selenium requiers a webdriver to be downloaded and put into a path, is it possible to use it from within web app? Like hosting the webdriver somewhere?
I am also open to solutions other than Selenium, if there are any.
I think what you would want is a selenium grid server.
https://www.seleniumhq.org/docs/07_selenium_grid.jsp
Basically you host it on some remote server and then you can connect to it and spin up web drivers remotely and use them in code as needed. It also comes with a handy interface for checking on current browser instances and even taking screenshots or executing scripts from the web ui.
I want to write an acceptance test for my browser extension. I've tried to initiate an extension via selenium but I can't seem to access a content of popup. Can someone suggest how can I do it with selenium or any other way to write UI/acceptance tests for browser extensions? Thanks.
How can I access browser extension's popup using selenium?
It's not possible. Selenium supports interaction with web view only.
What you can do with Selenium and extension for sure is automatic installation: https://stackoverflow.com/a/16512012/2517622
You may try to use desktop automation tools (e.g. White on Windows platform) for clicking on extension popup but it's not that easy and it's not platform independent as Selenium.
Here is the workaround we came up. Unless someone posts here the "right" solution I will consider this as the best approach.
So eventually, our extension is an iframe which just loads a page content from our website + does some other neat stuff. We simply open that url in a new tab and do regular selenium tests.
Side not: we have considered to write a little javascript wrapper to be able to access ext via main window through javascript. E.g. there is some js in ext that listens to main window's events and perform certain actions. Tho, it is too much efforts and doesn't really sound like a proper acceptance test so we discarded this approach.
There are many selenium webdriver binding package of Golang.
However, I don't want to control browser throught server.
How can I control browser with Golang and selenium without selenium server?
You can try github.com/fedesog/webdriver which says in its documentation:
This is a pure go library and doesn't require a running Selenium driver.
I would characterize the Selenium webdriver as a client rather than a server. Caveat: I have used the Selenium webdriver (Chrome version) from .Net and I am assuming it is similar for Go.
The way Selenium works is that you will launch an instance of it from within code, and it creates a live version of the selected browser (i.e. Chrome) and your program retains control over it. Then you write code to tell the browser to navigate to a page, inspect the response, and interact with the browser by filling out form data, clicking on buttons, etc. You can see what is happening on the browser as the code runs, so it is easy to troubleshoot when the interaction doesn't go as planned.
I have used Selenium to upload tens of thousands of records to a website that has no API and only a graphical user interface. Give it a chance.
I need to build a Python scraper to scrape data from a website where content is only displayed after a user clicks a link bound with a Javascript onclick function, and the page is not reloaded. I've looked into Selenium in order to do this and played around with it a bit, and it seems Selenium opens a new Firefox web browser everytime I instantiate a driver:
>>> driver = webdriver.Firefox()
Is this open browser required, or is there a way to get rid of it? I'm asking because the scraper is potentially part of a web app, and I'm afraid if multiple users start using it, I will have a bunch of browser windows open on my server.
Yes, selenium automates web browsers.
You can add this at the bottom of your python code to make sure the browser is closed at the end:
driver.quit()