There are many selenium webdriver binding package of Golang.
However, I don't want to control browser throught server.
How can I control browser with Golang and selenium without selenium server?
You can try github.com/fedesog/webdriver which says in its documentation:
This is a pure go library and doesn't require a running Selenium driver.
I would characterize the Selenium webdriver as a client rather than a server. Caveat: I have used the Selenium webdriver (Chrome version) from .Net and I am assuming it is similar for Go.
The way Selenium works is that you will launch an instance of it from within code, and it creates a live version of the selected browser (i.e. Chrome) and your program retains control over it. Then you write code to tell the browser to navigate to a page, inspect the response, and interact with the browser by filling out form data, clicking on buttons, etc. You can see what is happening on the browser as the code runs, so it is easy to troubleshoot when the interaction doesn't go as planned.
I have used Selenium to upload tens of thousands of records to a website that has no API and only a graphical user interface. Give it a chance.
Related
I am trying to scrape a website, but it is not loading in selenium. When I browse that website in my "real" chrome browser, everything works fine. Is there any way I can use my real browser with python to automate stuff, instead of using selenium??
Thanks
Using selenium we can automate real browsers.
If in case the website is not loading via selenium, you can check if adding desired capabilities helps.
Here we can set proxy, disable extensions etc. There are many options available.
https://chromedriver.chromium.org/capabilities
Also if you can share what kind of error is displayed that would be helpful.
I am building a web site in Django that would scrape data from some site, so people could enter the site, set custom data filters and view scraped data in friendly format.
The problem is that requests and beautiful soup modules will not be enough for the scraping purposes, since I will also need some automation to be done (loading javascript or clicking buttons).
Since Selenium requiers a webdriver to be downloaded and put into a path, is it possible to use it from within web app? Like hosting the webdriver somewhere?
I am also open to solutions other than Selenium, if there are any.
I think what you would want is a selenium grid server.
https://www.seleniumhq.org/docs/07_selenium_grid.jsp
Basically you host it on some remote server and then you can connect to it and spin up web drivers remotely and use them in code as needed. It also comes with a handy interface for checking on current browser instances and even taking screenshots or executing scripts from the web ui.
I am using htmlunit for web scraping - logging to a website on behalf of the users, settings something in their profile and then come back.
Just using pure Htmlunit and no selenium framework.
Now my question:
WebClient webClient = new WebClient(BrowserVersion.INTERNET_EXPLORER_11);
Does this statement - creates a browser instance on the machine where i am executing the code or what it does?
I am using BrowserVersion.INTERNET_EXPLORER_11 as this is an accepted browser at that website.
How Selenium is different than htmlunit - i know we can use htmlunit as a webdriver in Selenium. Does Selenium needs a native browser instance on the machine where the code is getting executed? Does Selenium creates browser instances?
My use case is - I will be having multiple users accessing this application. I know WebClient in htmlunit is not thread safe(so have to code it as Spring proto type bean).
Is there any suggestions regarding this?
Any help is greatly appreciated.
HTMLUnit is a headless browser. So no window will be created if used with Selenium either. Setting the BrowserVersion will just tell HTMLUnit to present itself to the server as if it were a given browser (AFAIK, it will just change the User-Agent but might perform additional internal processing depending on the version). I guess this answers most of the questions but the last one.
Regarding asking for suggestions on how to implement this I would try to avoid logging in to a website that way. If the website does not provide an API for this then it is likely that it is agains the Terms Of Service. Assuming it is not, you will have to create new WebClient instances for each user each time the data needs to be extracted from the other site.
I would like to make it clear about the difference between protractor VS ghostdriver.
With protractor:
start selenium web server for testing.
multiple browser testing.
whenever it start testing, it open the browser.
With ghostdriver:
start phantomjs web server.
can be config multiple browser too.
can run separate with selenium or integrate with selenium.
My question is PhantomJS webdriver can run alone without selenium webdriver, multiple browsers and good for CI. Why do we need to run selenium and integrate selenium with phantomjs using ghostdriver?
While I'm not entirely sure I understand your question, I'll take a stab at answering it. With WebDriver, driving a browser is done via a standardized JSON-over-HTTP wire protocol. This means that you need a "server" component that understands the wire protocol to drive any particular browser. For each of the major desktop browsers (Internet Explorer, Chrome, and Firefox), there is a server component that your WebDriver code talks to (IEDriverServer.exe, chromedriver.exe, or a Firefox browser extension, respectively). PhantomJS also implements a server component that understands the WebDriver wire protocol, so the same high-level WebDriver code can be used with PhantomJS that is used with other browsers. Note that the Selenium server is not required to drive any of the browsers on the local machine.
Now, since the protocol used is simply transmitted over HTTP, that gives WebDriver the opportunity to run the WebDriver code on one machine, while driving a browser located on an entirely different machine. That's where the Selenium server comes in. The Selenium server starts an HTTP server that understands the WebDriver JSON wire protocol. When that server receives a WebDriver command, it can forward that command to another "server" component, either running on that machine (as a standalone remote server), or on yet another machine running another instance of the Selenium server (in a "grid" configuration).
So to answer your question, yes, WebDriver code can be executed against PhantomJS without using the Selenium server. It can likewise be executed against Internet Explorer, Firefox, Chrome, Safari, and some versions of Opera, all without using the Selenium server. Notice that all of this is true without mentioning Protractor at all. Since Protractor is based on WebDriverJS, as long as there's a "server" component running, whether that's a Selenium server, chromedriver.exe, IEDriverServer.exe, or PhantomJS, the driver should be able to communicate with and drive that browser. Looking at the code, it appears that WebDriverJS (and, by extension, Protractor), should be able to execute against Chrome and PhantomJS without requiring the Selenium server, but I don't know enough about Protractor's wrapping of WebDriverJS to speak with authority.
I need to build a Python scraper to scrape data from a website where content is only displayed after a user clicks a link bound with a Javascript onclick function, and the page is not reloaded. I've looked into Selenium in order to do this and played around with it a bit, and it seems Selenium opens a new Firefox web browser everytime I instantiate a driver:
>>> driver = webdriver.Firefox()
Is this open browser required, or is there a way to get rid of it? I'm asking because the scraper is potentially part of a web app, and I'm afraid if multiple users start using it, I will have a bunch of browser windows open on my server.
Yes, selenium automates web browsers.
You can add this at the bottom of your python code to make sure the browser is closed at the end:
driver.quit()