Trying to run selenium script with targeted chrome profile. But once I run the script, it won't start with the targeted profile but with a new profile. Here's my code:
# import selenium common driver
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.keys import Keys
import time
# for specified chrome profile
from selenium.webdriver.chrome.options import Options
# wait page until targeted element loaded
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.common.exceptions import TimeoutException
# undetectable module
import undetected_chromedriver.v2 as uc # use pip install undetected-chromedriver
if __name__ == '__main__':
options = uc.ChromeOptions()
# another way to set profile is the below (which takes precedence if both variants are used
options.add_argument(r'--user-data-dir=C:\Users\Fadli\AppData\Local\Google\Chrome\User Data\Profile 4')
# just some options passing in to skip annoying popups
options.add_argument(r'--no-first-run --no-service-autorun --password-store=basic')
driver = uc.Chrome(options=options) # version_main allows to specify your chrome version instead of following chrome global version
driver.get('https://nowsecure.nl')
Try the below line if it help
# another way to set profile is the below (which takes precedence if both variants are used
options.add_argument(r'--user-data-dir=C:\Users\Fadli\AppData\Local\Google\Chrome\User Data')
options.add_argument(r'--profile-directory=Profile 4')
you can check it here for more reference
Related
So I'm trying to scrape some forecasting data from a website periodically, and ideally I would like for it to happen in the background. I had a look at some documentation and came up with the following code:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.chrome.options import Options
options = webdriver.ChromeOptions()
options.add_argument("--headless")
driver = webdriver.Chrome(options = options)
options.add_argument("disable-infobars")
options.add_argument("--disable-extensions")
driver = webdriver.Chrome()
driver.get('https://www.windguru.cz/53')
WebDriverWait(driver, 5).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "#forecasts-page")))
\#Scraping block of code goes here
driver.quit()
I think the following line is over-riding the --headless argument but i'm not sure.
WebDriverWait(driver, 5).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "#forecasts-page")))
The reason I have it added in the first place is that the website I'm scraping isn't just static html (have a look for yourself, link is in code). I think there's some js that prompts the forecast data to load, so I need to wait a bit and make sure before the script starts scraping the dom.
Any idea how I can achieve this and run the browser in headless mode?
The line that's opening the chrome is this
driver = webdriver.Chrome()
You can do away with this and code should still work the same.
How to use undetected_chromedriver.v2 in kali linux
import undetected_chromedriver.v2 as uc
options = uc.ChromeOptions()
# setting profile
options.user_data_dir = "c:\\temp\\profile"
# another way to set profile is the below (which takes precedence if both variants are used
options.add_argument('--user-data-dir=c:\\temp\\profile2')
# just some options passing in to skip annoying popups
options.add_argument('--no-first-run --no-service-autorun --password-store=basic')
bw = uc.Chrome(options=options, version_main=92)
If you are on selenium4 then using undetected_chromedriver.v2 you will face an error as:
TypeError: __init__() got an unexpected keyword argument 'service'
which is inline as per the status:
July 2021: Currently busy implementing selenium 4 for undetected-chromedriver
However with selenium4 you can still use the undetected_chromedriver v1 as follows:
import undetected_chromedriver as uc
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Service
options = uc.ChromeOptions()
options.add_argument("start-maximized")
s = Service('C:\\BrowserDrivers\\chromedriver.exe')
driver = uc.Chrome(service=s, options=options)
driver.get('https://datadome.co/customers-stories/toppreise-ends-web-scraping-and-content-theft-with-datadome/')
driver.save_screenshot('datadome_undetected_webddriver.png')
References
You can find a couple of relevant detailed discussions in:
Undetected Chromedriver not loading correctly
Evade detection of selenium automation
When the ChromeDriver version does not match my current chrome version, i upgrade chromedriver by the following code:
from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager
driver = webdriver.Chrome(ChromeDriverManager().install())
Then i use selenium for scraping the website data, but still i got some errors. Anyone can help me with this issue? Appreciate.
import time
import pandas as pd
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
options = Options()
options.headless = True
driver = webdriver.Chrome(options=options)
driver.get("https://www.binance.com/cn/futures/funding-history/0")
time.sleep(5)
The errors took place when the above code is ran which has been attached.
You may try to include the below code to get the latest version automatically
through PIP :
pip install chromedriver-autoinstaller
Usage
Just type import chromedriver_autoinstaller in the module you want to use chromedriver.
Example
from selenium import webdriver
import chromedriver_autoinstaller
chromedriver_autoinstaller.install() # Check if the current version of chromedriver exists
# and if it doesn't exist, download it automatically,
# then add chromedriver to path
driver = webdriver.Chrome()
driver.get("http://www.python.org")
assert "Python" in driver.title
Read more about auto upgrade here
Having some strangeness go on.
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.webdriver.common.action_chains import ActionChains
driver = None
class TestThreshold:
def __init__(self):
self.driver = webdriver.Chrome()
def waitForId(self,type,id):
try:
element_present = EC.presence_of_element_located((type,id))
WebDriverWait(self.driver, 10).until(element_present)
except TimeoutException:
print "Timed out waiting for page to load"
def setUp(self):
# code that uses driver.login to login
def tearDown(self):
self.driver.close();
def test_login(self):
# a test with assertion
def test_feature(self):
# a test with assertion
def test_admin(self):
# another test with assertion
When I run nosetests Chrome browser pops up. It stays on a blank page for a bit, then finally runs the test_login and test_feature then quits.
All 3 tests pass (got Ran 3 tests in CLI with an OK), but only 1 is visually shown. Two of the three ran in a headless type mode while the page was blank for a bit.
How can I have it run all tests from start to finish without headless? (And if I wanted to, how can i run them both headless?)
I also notice that if I keep adding tests, it always will run N-1 tests headless (one will be ran visually)
So this made me laugh. The reason is because there are multiple Chrome windows stacked on top of each other. It is running in parallel not sequentially.
Is there a command to run selenium tests without using a framework? e.g. pytest foo_test.py
What would be required on my local machine in order to run the following test? I am confused as this appears the only requirement would be chromedriver but I don't know which command to use in order to execute the actual test.
from selenium import webdriver
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
capa = DesiredCapabilities.CHROME
capa["pageLoadStrategy"] = "none"
driver = webdriver.Chrome(desired_capabilities=capa)
wait = WebDriverWait(driver, 20)
driver.get('http://stackoverflow.com/')
wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, '#h-top-questions')))
driver.execute_script("window.stop();")
Here is the Answer to your Question:
As you have asked Is there a command to run selenium tests without using a framework, the Answer is Yes.
To answer in simple words, there exists certain frameworks like pytest, unittest, etc in python to structure your test execution and interpreting the test results. Each of the frameworks have their own strengths. When the code base becomes bulky frameworks helps us to arrange. But using framework is not mandatory.
About your code, I don't see any significant error in your code but working with Selenium 3.x.x you need to download the chromedriver from here and save it in your machine. While you initialize the WebDriver instance you need to mention the absolute path of the chromedriver as below.
Here is your own code with some simple tweaks which works well at my end:
from selenium import webdriver
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
capa = DesiredCapabilities.CHROME
capa["pageLoadStrategy"] = "none"
driver = webdriver.Chrome(desired_capabilities=capa,executable_path="C:\\your_directory\\chromedriver.exe")
wait = WebDriverWait(driver, 20)
driver.get('http://stackoverflow.com/')
wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, '#h-top-questions')))
driver.execute_script("window.stop();")
Let me know if this Answers your Question.
There are actual 2 requirements that you are using. Selenium itself is a requirement, and then the chromedriver as you mentioned. The file is just a python file, so you can run it by doing python foo_test.py. There is also the option to use a framework like Unittest, which can be useful for seeing test results.
Selenium itself is not a "testing framework", it is a library of commands that allow a user to interact with a web browser. Selenium can be used for webscraping or automating tasks as well as testing purposes.