As the title, the following is not working on the digitalocean ubuntu droplet, but works on my local computer. All the software are the same.
It is just hanging there.
import os
from selenium import webdriver
from pyvirtualdisplay import Display
display = Display(visible=0, size=(800, 600))
display.start()
driver = webdriver.Chrome()
driver.get("http://www.google.com")
print driver.page_source.encode('utf-8')
driver.quit()
display.stop()
Can anybody give me some clue? Thanks a lot.I have searching all around and could not find any helpful information.
Installing Selenium and Chrome
apt install -y libxss1 libappindicator1 libindicator7
wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
dpkg -i google-chrome*.deb
apt install -y -f
pip3 install selenium
Check Google Chrome Version
google-chrome-stable -version
Install compatable chromedriver from https://chromedriver.chromium.org/downloads
wget https://chromedriver.storage.googleapis.com/77.0.3865.40/chromedriver_linux64.zip
apt install -y unzip
unzip chromedriver_linux64.zip
rm chromedriver_linux64.zip
chmod +x chromedriver
mv -f chromedriver /usr/local/bin/chromedriver
apt install -y xvfb
pip3 install pyvirtualdisplay
Then run your file!
Related
I tried to find the package for above said version for chrome from this link but i'm unable to find it. Where can i find the said package and what would be the steps to downgrade chrome on my Ubuntu 20.04 machine ?
First of all uninstall your current version of Chrome
$ sudo apt-get purge google-chrome-stable
$ mv ~/.config/google-chrome/ ~/.config/google-chrome.bak/
Then install the preferred version choosing from here and replacing the ${CHROME_VERSION} on the code below with the version you need (in this case 87.0.4280.88-1 )
wget --no-verbose -O /tmp/chrome.deb https://dl.google.com/linux/chrome/deb/pool/main/g/google-chrome-stable/google-chrome-stable_${CHROME_VERSION}_amd64.deb \
&& apt install -y /tmp/chrome.deb \
&& rm /tmp/chrome.deb
I am trying to setup jenkins using the official jenkins docker image.
Dockerfile
FROM jenkins/jenkins:lts
USER root
RUN apt-get update && apt install -y ruby-full
RUN apt-get install -y curl
RUN apt -y autoremove
RUN curl -sL https://deb.nodesource.com/setup_6.x | bash -
RUN apt-get install -y nodejs
RUN apt-get install -y build-essential
RUN apt install -y wget
RUN gem install --no-ri --no-rdoc --format-executable rake
RUN gem install selenium-webdriver
RUN gem install bundler
RUN npm install -g node-mongo-seeds
#Permissions granted to jenkins user to do a gem install
RUN chown -R jenkins:jenkins /var/lib/gems
RUN apt-get install -y patch ruby-dev zlib1g-dev liblzma-dev
RUN chown -R jenkins:jenkins /usr/local/bin
# We need wget to set up the PPA and xvfb to have a virtual screen and unzip to install the Chromedriver
RUN apt-get install -y wget xvfb unzip
# Set up the Chrome PPA
RUN wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add -
RUN echo "deb http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google.list
# Update the package list and install chrome
RUN apt-get update -y
RUN apt-get install -y google-chrome-stable
# Set up Chromedriver Environment variables
ENV CHROMEDRIVER_VERSION 2.35
ENV CHROMEDRIVER_DIR /chromedriver
RUN mkdir $CHROMEDRIVER_DIR
# Download and install Chromedriver
RUN wget -q --continue -P $CHROMEDRIVER_DIR "http://chromedriver.storage.googleapis.com/$CHROMEDRIVER_VERSION/chromedriver_linux64.zip"
RUN unzip $CHROMEDRIVER_DIR/chromedriver* -d $CHROMEDRIVER_DIR
# Put Chromedriver into the PATH
ENV PATH $CHROMEDRIVER_DIR:$PATH
USER jenkins
While running the job the browser won't open and hit the localhost.
I tried hitting 'google.com' as well
google-chrome --headless --no-sandbox 'https://www.google.com'
still no success
Fontconfig warning: "/etc/fonts/fonts.conf", line 100: unknown element "blank"
[0531/130727.949511:ERROR:browser_process_sub_thread.cc(217)] Waited 17 ms for network service
I am not able understand why jenkins user not able to open the browser.
Google Chrome 74.0.3729.169
ChromeDriver 2.35.528139 (47ead77cb35ad2a9a83248b292151462a66cd881)
ruby 2.3.3p222 (2016-11-21) [x86_64-linux-gnu]
Jenkins 2.164.3
This error message...
unknown error: Chrome failed to start: crashed (Selenium::WebDriver::Error::UnknownError)
...implies that the ChromeDriver was unable to initiate/spawn a new WebBrowser i.e. Chrome Browser session.
Your main issue is the incompatibility between the version of the binaries you are using as follows:
You are using chromedriver=2.35
Release Notes of chromedriver=2.35 clearly mentions the following :
Supports Chrome v62-64
You are using chrome=74.0
Release Notes of ChromeDriver v74.0 clearly mentions the following :
Supports Chrome v74
So there is a clear mismatch between ChromeDriver v2.35 and the Chrome Browser v74.0
Solution
Upgrade ChromeDriver to current ChromeDriver v74.0 level.
Keep Chrome version at Chrome v74 level. (as per ChromeDriver v2.46 release notes)
Clean your Project Workspace through your IDE and Rebuild your project with required dependencies only.
If your base Web Client version is too old, then uninstall it and install a recent GA and released version of Web Client.
Execute your #Test.
So as per #DebanjanB comment it resolved the driver issues but as per the new issue of chrome crashing I resolved it by just adding
options = Selenium::WebDriver::Chrome::Options.new
options.add_argument('--headless')
options.add_argument('--disable-gpu')
options.add_argument('--no-sandbox')
d = Selenium::WebDriver.for :chrome, options: options
I am trying to use selenium chrome driver with docker. Unfortunately I can't get it working. When I do docker-compose up to launch the container it crashes with a
"container_name exited with code 1" error.
Using docker logs -t -f I was able to get the below error.
selenium.common.exceptions.WebDriverException:
Message: unknown error: Chrome failed to start: exited abnormally
I know that there have been other stack overflow posts about this issue, but all the solutions involve adding chrome options. I have already tried adding various chrome driver options without any luck. See code below.
options = webdriver.ChromeOptions()
options.add_argument('--proxy-server=socks5://localhost:9050')
options.add_argument('disable-infobars')
options.add_argument('--disable-extensions')
options.add_argument('--no-sandbox')
options.add_argument('--disable-setuid-sandbox')
options.add_argument('--headless')
options.add_argument('--start-maximized')
options.add_argument('window-size=1200x800')
options.add_argument('--disable-gpu')
driver = webdriver.Chrome(options=options, desired_capabilities=caps)
I have tried various combinations of the above code with no luck. I also tried commenting out the 'proxy=server' option and running the container, but I get the same error. When I run the normal python code it works without any errors.
Below is the relevant portion of my Dockerfile.
# Extra suff chrome driver needs
RUN apt-get install -y libglib2.0-0=2.50.3-2 \
libnss3=2:3.26.2-1.1+deb9u1 \
libgconf-2-4=3.2.6-4+b1 \
libfontconfig1=2.11.0-6.7+b1
# need this for chrome driver, not just ui
RUN apt-get install -y wget xvfb unzip
# Set up the Chrome PPA
RUN wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add -
RUN echo "deb http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google.list
# Update the package list and install chrome
RUN apt-get update -y
RUN apt-get install -y google-chrome-stable
# Set up Chromedriver Environment variables
ENV CHROMEDRIVER_VERSION 2.19
ENV CHROMEDRIVER_DIR /chromedriver
RUN mkdir $CHROMEDRIVER_DIR
# Download and install Chromedriver
RUN wget -q --continue -P $CHROMEDRIVER_DIR "http://chromedriver.storage.googleapis.com/$CHROMEDRIVER_VERSION/chromedriver_linux64.zip"
RUN unzip $CHROMEDRIVER_DIR/chromedriver* -d $CHROMEDRIVER_DIR
# Put Chromedriver into the PATH
ENV PATH $CHROMEDRIVER_DIR:$PATH
You're facing an incompatibility between the chromedriver and chrome browser versions being used.
Here are the two lines in question:
RUN apt-get install -y google-chrome-stable
The above means that you'll download the latest chrome browser.
ENV CHROMEDRIVER_VERSION 2.19
However, over here you're specifically using ChromeDriver v2.19, which is a really old version.
You'll need to use compatible versions, which can generally be found in the release notes:
http://chromedriver.chromium.org/downloads
Ok so we downgraded Firefox 47 to previous stable version. BUT firefox keeps updating everyday to 47! Is there any way to completely stop firefox updates in Ubuntu server 14.04 (without GUI,only terminal) ?
We need versions <47 because Selenium has issues with Firefox version 47
Already Tried:
nano /usr/lib/firefox/defaults/pref/channel-prefs.js
nano /usr/lib/firefox/browser/defaults/preferences/nano syspref.js
nano /usr/lib/firefox/browser/defaults/preferences/user.js
commented out this: //pref("app.update.channel", "release"); and entered the following: pref("app.update.enabled", false); pref("app.update.silent", false);
I also created a profile folder in ~/.mozilla and added a text files with these: // turn off application updates: user_pref("app.update.enabled", false);
I also tried to block mozilla.org from iptables but that didn't help either.
To Hold firefox:
$ dpkg --get-selections | grep firefox
firefox install
firefox-locale-en install
firefox-locale-es install
unity-scope-firefoxbookmarks install
[13:17:03][root#robert]
[~]
$ echo firefox hold | sudo dpkg --set-selections
[13:17:20][root#robert]
[~]
$ dpkg --get-selections | grep firefox
firefox hold
firefox-locale-en install
firefox-locale-es install
unity-scope-firefoxbookmarks install
And to unhold:
[13:17:24][root#robert]
[~]
$ sudo apt-mark unhold firefox
Canceled hold on firefox.
[13:17:34][root#robert]
[~]
$ dpkg --get-selections | grep firefox
firefox install
firefox-locale-en install
firefox-locale-es install
unity-scope-firefoxbookmarks install
If your firefox is installed via dpkg or apt, you can do this:
echo "firefox hold" | sudo dpkg --set-selections
then verify the hold with :
dpkg --get-selections
And then you can try to update firefox with apt, normally it won't.
I want to use Scrapy which is a crawler framework, and install it to my Server.
This is my Server spec.
========================================================
NAME="Amazon Linux AMI"
VERSION="2015.09"
ID="amzn"
ID_LIKE="rhel fedora"
VERSION_ID="2015.09"
PRETTY_NAME="Amazon Linux AMI 2015.09"
ANSI_COLOR="0;33"
CPE_NAME="cpe:/o:amazon:linux:2015.09:ga"
HOME_URL="http://aws.amazon.com/amazon-linux-ami/"
Amazon Linux AMI release 2015.09
========================================================
I have read a lot of web pages about this and followed the steps.
Still I always get an error.
I've used the command sudo pip install scrapy and I get this log.
This is red logs.
-> Failed building wheel for lxml
-> Command "/usr/bin/python -c "import setuptools, tokenize;__file__='/tmp/pip-build-lT29Ha/lxml/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-WTCmji-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-lT29Ha/lxml
I've spent 2 days on the Scrapy install. Please help me.
I just did this yesterday on AWS. I used a nano instance so needed to add some swap first.
sudo fallocate -l 4G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo echo "/swapfile swap swap sw 0 0" >> /etc/fstab
sudo swapon /swapfile
Then install the prerequisites and scrapy:
sudo su
yum update -y
yum install python-pip -y
yum install python-devel -y
yum install gcc gcc-devel -y
yum install libxml2 libxml2-devel -y
yum install libxslt libxslt-devel -y
yum install openssl openssl-devel -y
yum install libffi libffi-devel -y
CFLAGS="-O0" pip install lxml
pip install scrapy
scrapy -v