When I run the spider from the Scrapy tutorial I get these error messages:
File "C:\Python26\lib\site-packages\twisted\internet\base.py", line 374, in fireEvent DeferredList(beforeResults).addCallback(self._continueFiring)
File "C:\Python26\lib\site-packages\twisted\internet\defer.py", line 195, in addCallback callbackKeywords=kw)
File "C:\Python26\lib\site-packages\twisted\internet\defer.py", line 186, in addCallbacks self._runCallbacks()
File "C:\Python26\lib\site-packages\twisted\internet\defer.py", line 328, in_runCallbacks self.result = callback(self.result, *args, **kw)
--- <exception caught here> ---
File "C:\Python26\lib\site-packages\twisted\internet\base.py", line 387, in _continueFiring callable(*args, **kwargs)
File "C:\Python26\lib\site-packages\twisted\internet\posixbase.py", line 356, in listenTCP p.startListening()
File "C:\Python26\lib\site-packages\twisted\internet\tcp.py", line 858, in startListening raise CannotListenError, (self.interface, self.port, le) twisted.internet.error.CannotListenError: Couldn't listen on any:6023: [Errno 10048]
Only one usage of each socket address (protocol/network address/port) is normally permitted.
Does anyone know what they are and how I can get rid of them?
Thanks
Perhaps you're running two Scrapy process simultaneously with telnet console enabled?.
If you want to run more than one Scrapy process at the same time, you must disable (or, at least, change the port) of web and telnet consoles.
Related
I need to use network namespaces in my setup, and it is working just fine.
However, when it comes to testing via seleniumLibrary to open a browser and test webinterface, i can't seem to get it to work with network namespaces.
I can manually run firefox in namespace by running firefox in the namespace, that has access the the network the namespace is assigned.
How do i do this in RobotFramework with seleniumLibrary?
I have this atm:
*** Settings ***
Documentation Test browser access in netns
Library OperatingSystem
Library SeleniumLibrary
Force Tags test_123
*** Test Cases ***
Test Browser In Netns
Open Browser 192.168.0.100 browser=ff
Close Browser
Running this produces the error: \
WebDriverException: Message: Service geckodriver unexpectedly exited. Status code was: 69
Traceback (most recent call last):
File "/home/mci/.local/lib/python3.8/site-packages/SeleniumLibrary/__init__.py", line 492, in run_keyword
return DynamicCore.run_keyword(self, name, args, kwargs)
File "/home/mci/.local/lib/python3.8/site-packages/robotlibcore.py", line 103, in run_keyword
return self.keywords[name](*args, **(kwargs or {}))
File "/home/mci/.local/lib/python3.8/site-packages/SeleniumLibrary/keywords/browsermanagement.py", line 293, in open_browser
return self._make_new_browser(
File "/home/mci/.local/lib/python3.8/site-packages/SeleniumLibrary/keywords/browsermanagement.py", line 324, in _make_new_browser
driver = self._make_driver(
File "/home/mci/.local/lib/python3.8/site-packages/SeleniumLibrary/keywords/browsermanagement.py", line 716, in _make_driver
driver = self._webdriver_creator.create_driver(
File "/home/mci/.local/lib/python3.8/site-packages/SeleniumLibrary/keywords/webdrivertools/webdrivertools.py", line 83, in create_driver
return creation_method(
File "/home/mci/.local/lib/python3.8/site-packages/SeleniumLibrary/keywords/webdrivertools/webdrivertools.py", line 199, in create_firefox
return webdriver.Firefox(
File "/home/mci/.local/lib/python3.8/site-packages/selenium/webdriver/firefox/webdriver.py", line 164, in __init__
self.service.start()
File "/home/mci/.local/lib/python3.8/site-packages/selenium/webdriver/common/service.py", line 98, in start
self.assert_process_still_running()
File "/home/mci/.local/lib/python3.8/site-packages/selenium/webdriver/common/service.py", line 109, in assert_process_still_running
raise WebDriverException(
Running this outside network namespace produces no errors, and goes through 100% pass.
Loopback is not automatically added and up'ed when creating netns.
This needs to be added with ip netns exec net-$name ip link set dev lo up.
GeckoDriver will try to bind itself to a random port on 127.0.0.1, so without lo being up, it will naturally fail.
I installed pgadmin4 (v4.4) package with from official Arch Linux package repository. It worked until the most recent update. Now I am not able to add a new server. In the browser I got the error:
http://127.0.0.1:36699/browser/server_group/children/1 500 (INTERNAL SERVER ERROR)
And from the server log, I got the error:
2019-04-09 21:41:49,900: ERROR flask.app: 'psycopg2.extensions.Column' object has no attribute '_asdict'
Traceback (most recent call last):
File "/usr/lib/python3.7/site-packages/flask/app.py", line 1813, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/lib/python3.7/site-packages/flask/app.py", line 1799, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/usr/lib/python3.7/site-packages/flask/views.py", line 88, in view
return self.dispatch_request(*args, **kwargs)
File "/usr/lib/pgadmin4/web/pgadmin/browser/utils.py", line 259, in dispatch_request
return method(*args, **kwargs)
File "/usr/lib/pgadmin4/web/pgadmin/browser/utils.py", line 309, in children
children.extend(module.get_nodes(*args, **kwargs))
File "/usr/lib/pgadmin4/web/pgadmin/browser/server_groups/servers/__init__.py", line 127, in get_nodes
in_recovery, wal_paused = recovery_state(conn, manager.version)
File "/usr/lib/pgadmin4/web/pgadmin/browser/server_groups/servers/__init__.py", line 52, in recovery_state
status, result = connection.execute_dict(recovery_check_sql)
File "/usr/lib/pgadmin4/web/pgadmin/utils/driver/psycopg2/connection.py", line 1202, in execute_dict
desc.to_dict() for desc in cur.ordered_description()
File "/usr/lib/pgadmin4/web/pgadmin/utils/driver/psycopg2/connection.py", line 1202, in <listcomp>
desc.to_dict() for desc in cur.ordered_description()
File "/usr/lib/pgadmin4/web/pgadmin/utils/driver/psycopg2/cursor.py", line 94, in to_dict
ores = OrderedDict(self.orig_col._asdict())
AttributeError: 'psycopg2.extensions.Column' object has no attribute '_asdict'
I wonder if whether it is only me because I didn't get anything from google search.
I have the same issue psycopg2 2.8.1. Downgrade to 2.7.7 makes the trick.
pgAdmin4 team has released the new version pgadmin4 (v4.5) which will work with latest version of psycopg2 which is psycopg2 2.8.1
https://www.pgadmin.org/download/
Bug: https://redmine.postgresql.org/issues/4143
In my case, first attempt of execution of command /usr/pgadmin4/bin/setup-web.sh showed below error that caused the web URL to show 500 error.
/usr/pgadmin4/bin/setup-web.sh: line 87: semanage: command not found
I'm using AlamLinux. I installed the required package and ran setup-web.sh again to make the web URL working
yum install policycoreutils-python-utils
/usr/pgadmin4/bin/setup-web.sh
I want to create a program in pythonista that can control the web browser. I know Selenium is the best for this but I have tried it on pythonista for my iOS iPhone and I get an error.
This is the code:
from selenium import webdriver
browser = webdriver.Chrome()
browser.get('http://www.yahoo.com')
Here is the error:
PermissionError: [Errno 1] Operation not permitted
Traceback (most recent call last):
File "/private/var/mobile/Containers/Shared/AppGroup/A2EBDF28-CB6C-4190-8199-7406AA3821A3/Pythonista3/Documents/selen.py", line 3, in <module>
browser = webdriver.Chrome()
File "/private/var/mobile/Containers/Shared/AppGroup/A2EBDF28-CB6C-4190-8199-7406AA3821A3/Pythonista3/Documents/site-packages-3/selenium/webdriver/chrome/webdriver.py", line 68, in __init__
self.service.start()
File "/private/var/mobile/Containers/Shared/AppGroup/A2EBDF28-CB6C-4190-8199-7406AA3821A3/Pythonista3/Documents/site-packages-3/selenium/webdriver/common/service.py", line 76, in start
stdin=PIPE)
File "/var/containers/Bundle/Application/24DD2A57-320E-4E21-9BE2-7C3605830DE0/Pythonista3.app/Frameworks/Py3Kit.framework/pylib/subprocess.py", line 708, in __init__
restore_signals, start_new_session)
File "/var/containers/Bundle/Application/24DD2A57-320E-4E21-9BE2-7C3605830DE0/Pythonista3.app/Frameworks/Py3Kit.framework/pylib/subprocess.py", line 1261, in _execute_child
restore_signals, start_new_session, preexec_fn)
PermissionError: [Errno 1] Operation not permitted
This error message...
PermissionError: [Errno 1] Operation not permitted
...implies that the ChromeDriver was unable to create a desired new resource e.g. logfile while initializing a new WebDriver and Web Client session.
As per the discussion Pythonista - Limitations due to iOS following are some of the limitations while using Pythonista :
No fork/exec for new processes. Impacts the subprocess module.
Due to missing fork, no full cleanup of process resources (memory, threads, file handles).
No file access outside of application directory.
No /dev/null and other special files.
Limited processing power of devices (compared to typical PC/Mac).
Process usually is stopped/killed after a while.
An simple example is as follows :
>>> import subprocess
>>> subprocess.call(["ls", "-l"])
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/private/var/mobile/Containers/Bundle/Application/8C59C68D-71BF-4CBB-90F8-373A1752DEE1/Pythonista.app/pylib/subprocess.py", line 524, in call
return Popen(*popenargs, **kwargs).wait()
File "/private/var/mobile/Containers/Bundle/Application/8C59C68D-71BF-4CBB-90F8-373A1752DEE1/Pythonista.app/pylib/subprocess.py", line 711, in __init__
errread, errwrite)
File "/private/var/mobile/Containers/Bundle/Application/8C59C68D-71BF-4CBB-90F8-373A1752DEE1/Pythonista.app/pylib/subprocess.py", line 1205, in _execute_child
self.pid = os.fork()
OSError: [Errno 1] Operation not permitted
What's wrong in your usecase
There can be 2 issues as follows :
When you invoke the following line of code :
browser = webdriver.Chrome()
The ChromeDriver tries to create/modify/access the scoped_directory within the file system. For example on Windows OS :
"chromedriverVersion": "2.35.528161 (5b82f2d2aae0ca24b877009200ced9065a772e73)",
"userDataDir": "C:\\Users\\username\\AppData\\Local\\Temp\\scoped_dir5188_12717"
Possibly ChromeDriver is unable to perform this task/method/functionality.
Again when you invoke the following line of code :
browser = webdriver.Chrome()
As per selenium.webdriver.chrome.webdriver ChromeDriver tries to create a logfile within the file system as per the constructor as follows :
class selenium.webdriver.chrome.webdriver.WebDriver(executable_path='chromedriver', port=0, options=None, service_args=None, desired_capabilities=None, service_log_path=None, chrome_options=None)
Possibly ChromeDriver is unable to perform this task/method/functionality,
Due to the above mentioned reasons you are seeing the error :
PermissionError: [Errno 1] Operation not permitted
Solution
Incase of any of the above mentioned cases the solution would be to restrict the access/creation of the resources within the application directory only.
When creating a managed session to use for distributed training with this line:
with sv.managed_session(server.target, config=config) as sess, sess.as_default():
I get this error (full stack trace at bottom) on the chief worker:
tensorflow.python.framework.errors_impl.UnavailableError: Trying to connect an http1.x server
Everything still seems to be fine on the parameter server which reports:
E1106 11:26:32.844686639 5543 ev_epoll1_linux.c:1051] grpc epoll fd: 8
2017-11-06 11:26:32.851773: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:215] Initialize GrpcChannelCache for job ps -> {0 -> localhost:12222}
2017-11-06 11:26:32.851863: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:215] Initialize GrpcChannelCache for job worker -> {0 -> 127.0.0.1:12223}
2017-11-06 11:26:32.856802: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:324] Started server with target: grpc://localhost:12222
I only receive this error when using the new v1.4 of tensorflow built from source (found same problem when installing from pip). Everything works fine in v1.3 . Does anyone know if there's been a breaking change made, I'm assuming with respect to how tensorflow works with grpc?
I'm wondering if this has something to do with http2 vs http1? I see GRPC seems to work with protobuf across http2, and this seems to be indicating its trying to connect with http1, but still doesn't explain why this breaks just when upgrading v1.3 to v1.4
Does anyone know any more around what that error
UnavailableError: Trying to connect an http1.x server
is referring to or what might be a fix here?
I am working on RedHat Linux and trying to do distributed training across processes on the same localhost...not even trying to go over the network. I'd appreciate any thoughts, and hope this can help others with the same problem as well.
Full stacktrace:
E1106 11:28:24.383745692 5787 ev_epoll1_linux.c:1051] grpc epoll fd: 8
2017-11-06 11:28:24.391084: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:215] Initialize
GrpcChannelCache for job ps -> {0 -> 127.0.0.1:12222}
2017-11-06 11:28:24.391185: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:215] Initialize
GrpcChannelCache for job worker -> {0 -> localhost:12223}
2017-11-06 11:28:24.392285: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:324] Started server
with target: grpc://localhost:12223
2017-11-06 11:28:37.875632: E tensorflow/core/distributed_runtime/master.cc:269] Master init: Unavailable:
Trying to connect an http1.x server
Traceback (most recent call last):
File "/app/sbtt/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1323, in
_do_call
return fn(*args)
File "/app/sbtt/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1293, in
_run_fn
self._extend_graph()
File "/app/sbtt/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1354, in
_extend_graph
self._session, graph_def.SerializeToString(), status)
File "/app/sbtt/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/errors_impl.py", line 473,
in __exit__
c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.UnavailableError: Trying to connect an http1.x server
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/pycharm-community-2017.2.3/helpers/pydev/pydevd.py", line 1599, in <module>
globals = debugger.run(setup['file'], None, None, is_module)
File "/opt/pycharm-community-2017.2.3/helpers/pydev/pydevd.py", line 1026, in run
pydev_imports.execfile(file, globals, locals) # execute the script
File "/opt/pycharm-community-2017.2.3/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
exec(compile(contents+"\n", file, 'exec'), glob, loc)
File "worker.py", line 426, in <module>
main()
File "worker.py", line 418, in main
run(args, server)
File "worker.py", line 174, in run
sess.run(trainer.sync)
File "/app/sbtt/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 889, in run
run_metadata_ptr)
File "/app/sbtt/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1120, in
_run
feed_dict_tensor, options, run_metadata)
File "/app/sbtt/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1317, in
_do_run
options, run_metadata)
File "/app/sbtt/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1336, in
_do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.UnavailableError: Trying to connect an http1.x server
if you follow #NoahEisen suggestion and
export GRPC_VERBOSITY="DEBUG"
you'll see something more informative like this:
E1108 17:37:57.085195825 17711 ev_epoll1_linux.c:1051] grpc epoll fd: 5
D1108 17:37:57.085309439 17711 ev_posix.c:111] Using polling engine: epoll1
D1108 17:37:57.085380147 17711 dns_resolver.c:301] Using native dns resolver
I1108 17:37:57.085819333 17711 socket_utils_common_posix.c:223] Disabling AF_INET6 sockets because ::1 is not available.
I1108 17:37:57.086001584 17711 tcp_server_posix.c:322] Failed to add :: listener, the environment may not support IPv6: {"created":"#1510180677.085876868","description":"OS Error","errno":97,"file":"external/grpc/src/core/lib/iomgr/socket_utils_common_posix.c","file_line":256,"os_error":"Address family not supported by protocol","syscall":"socket","target_address":"[::]:12223"}
2017-11-08 17:37:57.092525: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:215] Initialize GrpcChannelCache for job ps -> {0 -> 127.0.0.1:12222}
2017-11-08 17:37:57.092648: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:215] Initialize GrpcChannelCache for job worker -> {0 -> localhost:12223}
2017-11-08 17:37:57.093435: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:324] Started server with target: grpc://localhost:12223
D1108 17:38:02.607109518 17830 http_proxy.c:70] userinfo found in proxy URI
I1108 17:38:02.611335569 17807 http_connect_handshaker.c:304] Connecting to server 127.0.0.1:12222 via HTTP proxy ipv4:xx.xx.xx.xx:xxxx
2017-11-08 17:38:02.617814: E tensorflow/core/distributed_runtime/master.cc:269] Master init: Unavailable: Trying to connect an http1.x server
I am behind a proxy, but i am only trying to do distributed training on the localhost. For some reason it tries to connect via the proxy even tho the IP 127.0.0.1 should be equivalent to localhost right? IE note this part in particular:
Connecting to server 127.0.0.1:12222 via HTTP proxy ipv4:xx.xx.xx.xx:xxxx
I guess this was lazy in my python code. If I change the ps to "localhost" explicitly in the cluster spec instead of the IP 127.0.0.1 everything seems to work again in TF1.4 because its not trying to connect to the localhost via my proxy server (which indeed, was HTTP1.x only i think).
#PeteWaren - does this constitute an actual bug in tensorflow or grpc? Should these note be equivalent localhost=127.0.0.1? Either way, the way its handled has changed from TF1.3 to TF1.4
Thanks for everyones help
Recently when I tried to restart my buildmaster it failed because of some issues with http.log. I get the following error:
2014-07-17 11:21:55+0200 [-] RotateLogSite starting on 8000
2014-07-17 11:21:55+0200 [-] Starting factory <buildbot.status.web.baseweb.RotateLogSite instance at 0x0295F300>
2014-07-17 11:21:55+0200 [-] Setting up http.log rotating 10 files of 10000000 bytes each
2014-07-17 11:21:55+0200 [-] while starting BuildMaster
Traceback (most recent call last):
File "C:\Python27\lib\site-packages\twisted\application\service.py", line 308, in addService
service.privilegedStartService()
File "C:\Python27\lib\site-packages\twisted\application\service.py", line 277, in privilegedStartService
service.privilegedStartService()
File "C:\Python27\lib\site-packages\twisted\application\internet.py", line 358, in privilegedStartService
self._waitingForPort = self.endpoint.listen(self.factory)
File "C:\Python27\lib\site-packages\twisted\internet\endpoints.py", line 461, in listen
interface=self._interface)
--- <exception caught here> ---
File "C:\Python27\lib\site-packages\twisted\internet\defer.py", line 108, in execute
result = callable(*args, **kw)
File "C:\Python27\lib\site-packages\twisted\internet\posixbase.py", line 482, in listenTCP
p.startListening()
File "C:\Python27\lib\site-packages\twisted\internet\tcp.py", line 998, in startListening
self.factory.doStart()
File "C:\Python27\lib\site-packages\twisted\internet\protocol.py", line 72, in doStart
self.startFactory()
File "C:\Python27\lib\site-packages\twisted\web\http.py", line 1844, in startFactory
self.logFile = self._openLogFile(self.logPath)
File "C:\Python27\lib\site-packages\buildbot-0.8.8-py2.7.egg\buildbot\status\web\baseweb.py", line 477, in _openLogFile
return LogFile.fromFullPath(path, rotateLength=rotateLength, maxRotatedFiles=maxRotatedFiles)
File "C:\Python27\lib\site-packages\twisted\python\logfile.py", line 48, in fromFullPath
os.path.dirname(logPath), *args, **kwargs)
File "C:\Python27\lib\site-packages\twisted\python\logfile.py", line 161, in __init__
BaseLogFile.__init__(self, name, directory, defaultMode)
File "C:\Python27\lib\site-packages\twisted\python\logfile.py", line 40, in __init__
self._openFile()
File "C:\Python27\lib\site-packages\twisted\python\logfile.py", line 166, in _openFile
BaseLogFile._openFile(self)
File "C:\Python27\lib\site-packages\twisted\python\logfile.py", line 64, in _openFile
self._file = file(self.path, "r+", 1)
exceptions.IOError: [Errno 22] invalid mode ('r+') or filename: 'E:\\master_dir\\http.log'
I found that my http.log has reached a size of 4GB although it should rotate after 10MB. Why the buildbot was not able to rotate the http.log after 10MB? Why I am getting the error Invalid mode (r+)? Is there a build config parameter to set the rotate size for http.log?
Deleting the file atleast solved the problem of starting buildmaster but I am still clueless why I got this error at the very first place.
I am running buildbot version 0.8.8
Thanks.
EDIT
As asked by djmitche, I tried with twisted logging module. It was logging fine, irrespective of the order of slashes.
######## TWISTED ##########
from twisted.python.logfile import LogFile
LOG_FILENAME_TWISTED = r'C:\wp\log_rotate_twisted.log'
lf = LogFile.fromFullPath(LOG_FILENAME_TWISTED, rotateLength=20)
# Log some messages
for i in range(20):
lf.write('i = %d' % i)
This may be an issue with Twisted Python (and I've proposed adding the corresponding tag).
Can you verify that the log rotation parameters are set correctly, perhaps by patching C:\Python27\lib\site-packages\buildbot-0.8.8-py2.7.egg\buildbot\status\web\baseweb.py in _openLogFile to print rotateLength? The printed output will appear in twistd.log when you restart the master.