I am trying to send request with proxies, the problem is, if I use IP:Port proxies, my script work fine, but if I use IP:Port which ask for username and password too, that case my script unable to connect website.
How can I set up proxies properly with Firefox Selenium?
Here is the code:
import time
from stem import Signal
from stem.control import Controller
from selenium.webdriver import Firefox
from selenium import webdriver
from selenium.webdriver.firefox.service import Service
from selenium.webdriver.firefox.options import Options
# profile_path = r'C:\Users\hp\Desktop\temp\FirefoxPortable\Data'
options=Options()
# options.set_preference('profile', profile_path)
options.set_preference('network.proxy.type', 1)
options.set_preference('network.proxy.socks', 'myhostgoes here')
options.set_preference('network.proxy.socks_port',12323)
options.set_preference("network.proxy.socks_username",'myusername goes here')
options.set_preference("network.proxy.socks_password",'Password goes here')
options.set_preference('network.proxy.socks_remote_dns', True)
service = Service(r'C://SeleniumDrivers/geckodriver.exe')
driver = Firefox(service=service, options=options)
driver.get("https://www.whoer.net")
time.sleep(15)
driver.close()
driver = Firefox(service=service, options=options)
driver.get("https://www.whoer.net")
Related
I tried to launch chrome with selenium. but as soon as the browser loads the urls, google chrome closes automatically. here is the code:
'''
from selenium import webdriver
url= 'https://www.gmail.com'
from webdriver_manager.chrome import ChromeDriverManager
driver = webdriver.Chrome(ChromeDriverManager().install())
driver.get(url)
'''
Because selenium4 no need to use driver.close()
method they automatically close the driver after execution. So options.add_experimental_option("detach", True) argument will help you to prevent to close immediately after once launch the url with google chrome
# selenium 4
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager
import time
options = webdriver.ChromeOptions()
options.add_argument("start-maximized")
#chrome to stay open
options.add_experimental_option("detach", True)
driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()),options=options)
url= 'https://www.gmail.com'
driver.get(url)
time.sleep(2)
I had Selenium in an previous version, and installed Selenium 4.2.0, and my code is not working:
from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.chrome.service import Service
servico = Service(ChromeDriverManager().install())
navegador = webdriver.Chrome(service=servico)
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.chrome.options import Options
import time
from selenium.common.exceptions.WebDriverException import NoSuchElementException
options = Options()
options.headless = False
navegador.get("https://receita.pr.gov.br/")
print ("Headless Chrome Initialized")
# entry with cpf
navegador.find_element(By.XPATH, '//*[#id="cpfusuario"]').send_keys("000000000")
I uninstalled Selenium and reinstalled, also reinstalled Chrome Driver, but I don't know why it isn't working.
There is no error message.
In your code, you initialized options after you already launched the driver. That should be done before you launch the driver so that you can use those options with any other argument that you want to pass in, such as the service:
driver = webdriver.Chrome(options=options)
# Or in the case of your example with translation:
navegador = webdriver.Chrome(service=servico, options=options)
I am adding chrome options this way and it works if I use proxy ip authentication.
options = webdriver.ChromeOptions()
options.headless = True
options.add_argument('--proxy-server=92.128.165.143:3399')
driver = uc.Chrome(options=options)
However, I have a proxy with authentication in this format:
http://username:password#91.92.128.165.143:3399
If I add it like
options.add_argument('--proxy-server=http://username:password#91.92.128.165.143:3399')
it doesn't work. How could I add it with username/password? This applies only to undetected chrome driver.
I think Ive achieved it already by the help of selenium wire but it didnt work with kivy gui so for scripting you can carryon like this but if you wanna use with kivy then definitely you will get error
from ast import Try
from pathlib import Path
from selenium.webdriver.common.by import By
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.proxy import *
from seleniumwire import undetected_chromedriver as uc
from fake_useragent import UserAgent
from selenium.webdriver.common.action_chains import ActionChains
from selenium.webdriver.common.keys import Keys
from pyclick import HumanClicker
import pyautogui
import time
clicked=False
hostname = "188.74.183.126"
port = "8395"
proxy_username = "wclmiemy"
proxy_password = "a9hoxl4phkzr"
chrome_options = {
'proxy': {
'http': f'http://{proxy_username}:{proxy_password}#{hostname}:{port}',
'https': f'https://{proxy_username}:{proxy_password}#{hostname}:{port}',
'no_proxy': 'localhost,127.0.0.1'
}
}
def delete_cache(driver):
driver.execute_script("window.open('')") # Create a separate tab than the main one
driver.switch_to.window(driver.window_handles[-1]) # Switch window to the second tab
driver.get('chrome://settings/clearBrowserData') # Open your chrome settings.
pyautogui.click("clear_data.png")
if __name__ == '__main__':
email = "moqaddasmehran5#gmail.com"
password = "moqaddaszaheenzaheen"
ua = UserAgent()
userAgent = ua.random
options = webdriver.ChromeOptions()
options.add_argument(f'user-agent={userAgent}')
# options.add_argument('--ignore-certificate-errors-spki-list')
# # options.add_argument('--ignore-ssl-errors')
options.add_argument("--disable-infobars")
browser = uc.Chrome(
driver_executable_path="chromedriver",
seleniumwire_options=chrome_options,
options=options,
use_subprocess=True
)
browser.maximize_window()
browser.get('https://www.youtube.com/watch?v=uPxkrGL0l7U')
```
this code is kind of messy but im in Hurry hope you will be able to modify you willa also get ssle that you need to import in chrome thats it you will defnitely get it
Use the following code to add proxy with username and password:
from selenium import webdriver
PROXY = "http://username:password#91.92.128.165.143:3399"
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument('--proxy-server=%s' % PROXY)
driver = webdriver.Chrome(chrome_options=chrome_options)
edit:
I found this how to set proxy with authentication in selenium chromedriver python?
I am trying to scrape this website using python's BeautifulSoup package and for automating the user flow I am using selenium. As this website requires authentication to access this page, I am trying to log in first using selenium webdriver. Here is my code:
from bs4 import BeautifulSoup
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import WebDriverWait
from selenium.common.exceptions import TimeoutException
def configure_driver():
# Add additional Options to the webdriver
chrome_options = Options()
# add the argument and make the browser Headless.
chrome_options.add_argument("--headless")
# Instantiate the Webdriver: Mention the executable path of the webdriver you have downloaded
# For linux/Mac
# driver = webdriver.Chrome(options = chrome_options)
# For windows
driver = webdriver.Chrome(executable_path="/home/<user_name>/Downloads/chromedriver_linux64/chromedriver",
options = chrome_options)
return driver
def getLinks(driver):
# Step 1: Go to pluralsight.com, category section with selected search keyword
driver.get(f"https://www.coursera.org/learn/competitive-data-science/supplement/RrrDR/additional-material-and-links")
# wait for the element to load
try:
WebDriverWait(driver, 5).until(lambda s: s.find_element_by_class_name("_ojjigd").is_displayed())
except TimeoutException:
print("TimeoutException: Element not found")
return None
email = driver.find_element_by_name('email')
print(str(email))
password = driver.find_element_by_name('password')
email.send_keys("username") # provide some actual username
password.send_keys("password") # provide some actual password
form = driver.find_element_by_name('login')
print(form.submit())
WebDriverWait(driver, 10)
print(driver.title)
soup = BeautifulSoup(driver.page_source)
# Step 3: Iterate over the search result and fetch the course
divs = soup.findAll('div', args={'class': 'item-box-content'})
print(len(divs))
# create the driver object.
driver = configure_driver()
getLinks(driver)
# close the driver.
driver.close()
Now after doing form.submit() it is expected to log in and change the page, right? But it is simply staying in the same page, so I cannot access the contents of the authenticated page. Someone please help.
That is because there is no name attribute.
instead of this :
form = driver.find_element_by_name('login')
Use this :
wait.until(EC.element_to_be_clickable((By.XPATH, "//button[text()='Login']"))).click()
I tried this code on my local, seems to be working fine
driver.maximize_window()
wait = WebDriverWait(driver, 30)
driver.get("https://www.coursera.org/learn/competitive-data-science/supplement/RrrDR/additional-material-and-links")
wait.until(EC.element_to_be_clickable((By.ID, "email"))).send_keys("some user name")
wait.until(EC.element_to_be_clickable((By.ID, "password"))).send_keys("some user name")
wait.until(EC.element_to_be_clickable((By.XPATH, "//button[text()='Login']"))).click()
Since login button is in a form so .submit() should work too.
wait.until(EC.element_to_be_clickable((By.XPATH, "//button[text()='Login']"))).submit()
This works too.
I'm using splinter(v0.7.3) for web testing under linux, while on chrome, the default sample code can not running:
from splinter import Browser
from pyvirtualdisplay import Display
d = Display(visible=0, size=(800, 600))
d.start()
b = Browser('chrome')
b.visit('http://www.google.com')
b.quit()
d.stop()
While running, I got the exception like this:
selenium.common.exceptions.WebDriverException: Message: chrome not reachable
And I test the same function in selenium with some chrome option added:
from selenium import web driver
from selenium.webdriver.chrome.options import Options
from pyvirtualdisplay import Display
d = Display(visible=0, size=(800, 600))
d.start()
opt = Options()
opt.add_argument('--disable-setuid-sandbox')
b = webdriver.Chrome(chrome_options=opt)
b.get('http://www.google.com')
b.quit()
d.stop()
This works ok, the difference is the --disable-setuid-sandbox option added to chrome driver, if the option not add, there would be a zombie chrome-sandbox process under chromium-browser.
The problem here is, I don't know how to pass a chrome.options.Option instance to splinter.Browser(), I browsed the implementation under splinter/driver/webdriver/chrome.py, it seems that there is no entry to pass such a instance to splinter.Browser(). Is there some other way to pass options to chrome driver?
Create a new instance of BaseWebDriver and set .driver with an instance of the Chrome driver. This example starts Chrome maximized:
from selenium.webdriver import Chrome
from selenium.webdriver.chrome.options import Options
from splinter.driver.webdriver import BaseWebDriver, WebDriverElement
options = Options()
options.add_argument('--start-maximized')
browser = BaseWebDriver()
browser.driver = Chrome(chrome_options=options)
browser.visit('https://www.google.com')
The only way I could ever do this was by using the add_argument method with selenium.webdriver.ChromeOptions like so:
from selenium.webdriver import ChromeOptions
from splinter import Browser
chrome_options = ChromeOptions()
chrome_options.add_argument(your_argument)
b=Browser("chrome", options=chrome_options)
b.visit('http://www.google.com')
b.quit()
so in your code would be:
from splinter import Browser
from selenium.webdriver import ChromeOptions
from pyvirtualdisplay import Display #I'm not certain what this is...
d = Display(visible=0, size=(800, 600))
d.start()
chrome_options = ChromeOptions()
chrome_options.add_argument('disable-setuid-sandbox')
b = Browser('chrome')
b.visit('http://www.google.com')
b.quit()
d.stop()
Note: I was unable to test this with your argument specifically because I recently broke my GRUB so I am stuck in windows, and the disable-setuid-sandbox option is linux-only. However, I have been using this method with the headless argument for a while.
I am not 100% sure that this will work but I just looked at the docs for splinter and it says.
You can also pass additional arguments that correspond to Selenium DesiredCapabilities arguments.
Looking into the sourcecode of Splinter calling Browser can take some arguments. These arguments will then be passed to create an Instance of the Chrome WebDriver. So I went to the selenium sourcecode and saw the constructor looks like this:
def __init__(self, executable_path="chromedriver", port=0,
chrome_options=None, service_args=None,
desired_capabilities=None, service_log_path=None):
There is a parameter for chrome_options so it should be possible to pass it using this parameter. So if I'm correct this should work fine:
opt = Options()
opt.add_argument('--disable-setuid-sandbox')
b = Browser(browser='chrome', chrome_options=opt)
Edit
Alternatively you could pass the options as desired capabilities aswell:
opt = Options()
opt.add_argument('--disable-setuid-sandbox')
dc = opt.to_capabilities()
b = Browser(browser='chrome', desired_capabilities=dc)
I've been working on a fork of splinter for the past couple weeks, you can check out my dev branch if you want. I have added this and other features.
Options can be passed as a list to the chrome_options parameter
from splinter import Browser
options = ['--start-maximized', '--disable-setuid-sandbox']
with Browser('chrome', chrome_options=options) as browser:
browser.visit('http://www.google.com')
Edit:
So it turns out this was possible with splinter all along just using **kwargs which passes the various available options to the selenium driver(s). For example:
from splinter import Browser
options = {'chrome_options':['--start-maximized', '--disable-setuid-sandbox']}
with Browser('chrome', **options) as browser:
browser.visit('http://www.google.com')