Buildbot master's http.log error - twisted

Recently when I tried to restart my buildmaster it failed because of some issues with http.log. I get the following error:
2014-07-17 11:21:55+0200 [-] RotateLogSite starting on 8000
2014-07-17 11:21:55+0200 [-] Starting factory <buildbot.status.web.baseweb.RotateLogSite instance at 0x0295F300>
2014-07-17 11:21:55+0200 [-] Setting up http.log rotating 10 files of 10000000 bytes each
2014-07-17 11:21:55+0200 [-] while starting BuildMaster
Traceback (most recent call last):
File "C:\Python27\lib\site-packages\twisted\application\service.py", line 308, in addService
service.privilegedStartService()
File "C:\Python27\lib\site-packages\twisted\application\service.py", line 277, in privilegedStartService
service.privilegedStartService()
File "C:\Python27\lib\site-packages\twisted\application\internet.py", line 358, in privilegedStartService
self._waitingForPort = self.endpoint.listen(self.factory)
File "C:\Python27\lib\site-packages\twisted\internet\endpoints.py", line 461, in listen
interface=self._interface)
--- <exception caught here> ---
File "C:\Python27\lib\site-packages\twisted\internet\defer.py", line 108, in execute
result = callable(*args, **kw)
File "C:\Python27\lib\site-packages\twisted\internet\posixbase.py", line 482, in listenTCP
p.startListening()
File "C:\Python27\lib\site-packages\twisted\internet\tcp.py", line 998, in startListening
self.factory.doStart()
File "C:\Python27\lib\site-packages\twisted\internet\protocol.py", line 72, in doStart
self.startFactory()
File "C:\Python27\lib\site-packages\twisted\web\http.py", line 1844, in startFactory
self.logFile = self._openLogFile(self.logPath)
File "C:\Python27\lib\site-packages\buildbot-0.8.8-py2.7.egg\buildbot\status\web\baseweb.py", line 477, in _openLogFile
return LogFile.fromFullPath(path, rotateLength=rotateLength, maxRotatedFiles=maxRotatedFiles)
File "C:\Python27\lib\site-packages\twisted\python\logfile.py", line 48, in fromFullPath
os.path.dirname(logPath), *args, **kwargs)
File "C:\Python27\lib\site-packages\twisted\python\logfile.py", line 161, in __init__
BaseLogFile.__init__(self, name, directory, defaultMode)
File "C:\Python27\lib\site-packages\twisted\python\logfile.py", line 40, in __init__
self._openFile()
File "C:\Python27\lib\site-packages\twisted\python\logfile.py", line 166, in _openFile
BaseLogFile._openFile(self)
File "C:\Python27\lib\site-packages\twisted\python\logfile.py", line 64, in _openFile
self._file = file(self.path, "r+", 1)
exceptions.IOError: [Errno 22] invalid mode ('r+') or filename: 'E:\\master_dir\\http.log'
I found that my http.log has reached a size of 4GB although it should rotate after 10MB. Why the buildbot was not able to rotate the http.log after 10MB? Why I am getting the error Invalid mode (r+)? Is there a build config parameter to set the rotate size for http.log?
Deleting the file atleast solved the problem of starting buildmaster but I am still clueless why I got this error at the very first place.
I am running buildbot version 0.8.8
Thanks.
EDIT
As asked by djmitche, I tried with twisted logging module. It was logging fine, irrespective of the order of slashes.
######## TWISTED ##########
from twisted.python.logfile import LogFile
LOG_FILENAME_TWISTED = r'C:\wp\log_rotate_twisted.log'
lf = LogFile.fromFullPath(LOG_FILENAME_TWISTED, rotateLength=20)
# Log some messages
for i in range(20):
lf.write('i = %d' % i)

This may be an issue with Twisted Python (and I've proposed adding the corresponding tag).
Can you verify that the log rotation parameters are set correctly, perhaps by patching C:\Python27\lib\site-packages\buildbot-0.8.8-py2.7.egg\buildbot\status\web\baseweb.py in _openLogFile to print rotateLength? The printed output will appear in twistd.log when you restart the master.

Related

pgadmin4 showing 500 (INTERNAL SERVER ERROR)

I installed pgadmin4 (v4.4) package with from official Arch Linux package repository. It worked until the most recent update. Now I am not able to add a new server. In the browser I got the error:
http://127.0.0.1:36699/browser/server_group/children/1 500 (INTERNAL SERVER ERROR)
And from the server log, I got the error:
2019-04-09 21:41:49,900: ERROR flask.app: 'psycopg2.extensions.Column' object has no attribute '_asdict'
Traceback (most recent call last):
File "/usr/lib/python3.7/site-packages/flask/app.py", line 1813, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/lib/python3.7/site-packages/flask/app.py", line 1799, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/usr/lib/python3.7/site-packages/flask/views.py", line 88, in view
return self.dispatch_request(*args, **kwargs)
File "/usr/lib/pgadmin4/web/pgadmin/browser/utils.py", line 259, in dispatch_request
return method(*args, **kwargs)
File "/usr/lib/pgadmin4/web/pgadmin/browser/utils.py", line 309, in children
children.extend(module.get_nodes(*args, **kwargs))
File "/usr/lib/pgadmin4/web/pgadmin/browser/server_groups/servers/__init__.py", line 127, in get_nodes
in_recovery, wal_paused = recovery_state(conn, manager.version)
File "/usr/lib/pgadmin4/web/pgadmin/browser/server_groups/servers/__init__.py", line 52, in recovery_state
status, result = connection.execute_dict(recovery_check_sql)
File "/usr/lib/pgadmin4/web/pgadmin/utils/driver/psycopg2/connection.py", line 1202, in execute_dict
desc.to_dict() for desc in cur.ordered_description()
File "/usr/lib/pgadmin4/web/pgadmin/utils/driver/psycopg2/connection.py", line 1202, in <listcomp>
desc.to_dict() for desc in cur.ordered_description()
File "/usr/lib/pgadmin4/web/pgadmin/utils/driver/psycopg2/cursor.py", line 94, in to_dict
ores = OrderedDict(self.orig_col._asdict())
AttributeError: 'psycopg2.extensions.Column' object has no attribute '_asdict'
I wonder if whether it is only me because I didn't get anything from google search.
I have the same issue psycopg2 2.8.1. Downgrade to 2.7.7 makes the trick.
pgAdmin4 team has released the new version pgadmin4 (v4.5) which will work with latest version of psycopg2 which is psycopg2 2.8.1
https://www.pgadmin.org/download/
Bug: https://redmine.postgresql.org/issues/4143
In my case, first attempt of execution of command /usr/pgadmin4/bin/setup-web.sh showed below error that caused the web URL to show 500 error.
/usr/pgadmin4/bin/setup-web.sh: line 87: semanage: command not found
I'm using AlamLinux. I installed the required package and ran setup-web.sh again to make the web URL working
yum install policycoreutils-python-utils
/usr/pgadmin4/bin/setup-web.sh

PermissionError: [Errno 1] Operation not permitted while using Selenium with Pythonista on iOS

I want to create a program in pythonista that can control the web browser. I know Selenium is the best for this but I have tried it on pythonista for my iOS iPhone and I get an error.
This is the code:
from selenium import webdriver
browser = webdriver.Chrome()
browser.get('http://www.yahoo.com')
Here is the error:
PermissionError: [Errno 1] Operation not permitted
Traceback (most recent call last):
File "/private/var/mobile/Containers/Shared/AppGroup/A2EBDF28-CB6C-4190-8199-7406AA3821A3/Pythonista3/Documents/selen.py", line 3, in <module>
browser = webdriver.Chrome()
File "/private/var/mobile/Containers/Shared/AppGroup/A2EBDF28-CB6C-4190-8199-7406AA3821A3/Pythonista3/Documents/site-packages-3/selenium/webdriver/chrome/webdriver.py", line 68, in __init__
self.service.start()
File "/private/var/mobile/Containers/Shared/AppGroup/A2EBDF28-CB6C-4190-8199-7406AA3821A3/Pythonista3/Documents/site-packages-3/selenium/webdriver/common/service.py", line 76, in start
stdin=PIPE)
File "/var/containers/Bundle/Application/24DD2A57-320E-4E21-9BE2-7C3605830DE0/Pythonista3.app/Frameworks/Py3Kit.framework/pylib/subprocess.py", line 708, in __init__
restore_signals, start_new_session)
File "/var/containers/Bundle/Application/24DD2A57-320E-4E21-9BE2-7C3605830DE0/Pythonista3.app/Frameworks/Py3Kit.framework/pylib/subprocess.py", line 1261, in _execute_child
restore_signals, start_new_session, preexec_fn)
PermissionError: [Errno 1] Operation not permitted
This error message...
PermissionError: [Errno 1] Operation not permitted
...implies that the ChromeDriver was unable to create a desired new resource e.g. logfile while initializing a new WebDriver and Web Client session.
As per the discussion Pythonista - Limitations due to iOS following are some of the limitations while using Pythonista :
No fork/exec for new processes. Impacts the subprocess module.
Due to missing fork, no full cleanup of process resources (memory, threads, file handles).
No file access outside of application directory.
No /dev/null and other special files.
Limited processing power of devices (compared to typical PC/Mac).
Process usually is stopped/killed after a while.
An simple example is as follows :
>>> import subprocess
>>> subprocess.call(["ls", "-l"])
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/private/var/mobile/Containers/Bundle/Application/8C59C68D-71BF-4CBB-90F8-373A1752DEE1/Pythonista.app/pylib/subprocess.py", line 524, in call
return Popen(*popenargs, **kwargs).wait()
File "/private/var/mobile/Containers/Bundle/Application/8C59C68D-71BF-4CBB-90F8-373A1752DEE1/Pythonista.app/pylib/subprocess.py", line 711, in __init__
errread, errwrite)
File "/private/var/mobile/Containers/Bundle/Application/8C59C68D-71BF-4CBB-90F8-373A1752DEE1/Pythonista.app/pylib/subprocess.py", line 1205, in _execute_child
self.pid = os.fork()
OSError: [Errno 1] Operation not permitted
What's wrong in your usecase
There can be 2 issues as follows :
When you invoke the following line of code :
browser = webdriver.Chrome()
The ChromeDriver tries to create/modify/access the scoped_directory within the file system. For example on Windows OS :
"chromedriverVersion": "2.35.528161 (5b82f2d2aae0ca24b877009200ced9065a772e73)",
"userDataDir": "C:\\Users\\username\\AppData\\Local\\Temp\\scoped_dir5188_12717"
Possibly ChromeDriver is unable to perform this task/method/functionality.
Again when you invoke the following line of code :
browser = webdriver.Chrome()
As per selenium.webdriver.chrome.webdriver ChromeDriver tries to create a logfile within the file system as per the constructor as follows :
class selenium.webdriver.chrome.webdriver.WebDriver(executable_path='chromedriver', port=0, options=None, service_args=None, desired_capabilities=None, service_log_path=None, chrome_options=None)
Possibly ChromeDriver is unable to perform this task/method/functionality,
Due to the above mentioned reasons you are seeing the error :
PermissionError: [Errno 1] Operation not permitted
Solution
Incase of any of the above mentioned cases the solution would be to restrict the access/creation of the resources within the application directory only.

Tensorflow could not initialize the libcurl library on Mac OS

I have build up the WebAPP of this Project on Mac OS using conda and tensorflow v0.12.1. It work so well still I try to train, tensorflow show this error message:
W tensorflow/core/platform/cloud/google_auth_provider.cc:151] All
attempts to get a Google authentication bearer token failed, returning
an empty token. Retrieving token from files failed with "Failed
precondition: Could not initialize the libcurl library. Please make
sure that libcurl is installed in the OS or statically linked to the
TensorFlow binary.". Retrieving token from GCE failed with "Failed
precondition: Could not initialize the libcurl library. Please make
sure that libcurl is installed in the OS or statically linked to the
TensorFlow binary.".
Logs:
2018-04-03 09:33:49,154 - candysorter.views.api - INFO - === Start training: id=9120093671565748, session=20180403_093211_9120093671565748 ===
2018-04-03 09:33:49,154 - candysorter.views.api - INFO - Creating labels file: job_id=candy_sorter_20180403_093211_9120093671565748
2018-04-03 09:33:49,184 - candysorter.views.api - ERROR - Unexpected error.
Traceback (most recent call last):
File "/Users/wubinbin/anaconda3/envs/candy/lib/python2.7/site-packages/flask/app.py", line 1612, in full_dispatch_request
rv = self.dispatch_request()
File "/Users/wubinbin/anaconda3/envs/candy/lib/python2.7/site-packages/flask/app.py", line 1598, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/Users/wubinbin/Developer/FindYourCandy/webapp/candysorter/views/api.py", line 101, in wrapper
return f(*args, **kwargs)
File "/Users/wubinbin/Developer/FindYourCandy/webapp/candysorter/views/api.py", line 337, in train
candy_trainer.create_labels_file(job_id, labels)
File "/Users/wubinbin/Developer/FindYourCandy/webapp/candysorter/models/images/train.py", line 76, in create_labels_file
f.write(json.dumps(labels, separators=(',', ':')))
File "/Users/wubinbin/anaconda3/envs/candy/lib/python2.7/site-packages/tensorflow/python/lib/io/file_io.py", line 150, in __exit__
self.close()
File "/Users/wubinbin/anaconda3/envs/candy/lib/python2.7/site-packages/tensorflow/python/lib/io/file_io.py", line 182, in close
pywrap_tensorflow.Set_TF_Status_from_Status(status, ret_status)
File "/Users/wubinbin/anaconda3/envs/candy/lib/python2.7/contextlib.py", line 24, in __exit__
self.gen.next()
File "/Users/wubinbin/anaconda3/envs/candy/lib/python2.7/site-packages/tensorflow/python/framework/errors_impl.py", line 469, in raise_exception_on_not_ok_status
pywrap_tensorflow.TF_GetCode(status))
FailedPreconditionError: Could not initialize the libcurl library. Please make sure that libcurl is installed in the OS or statically linked to the TensorFlow binary.
I use TF1.7.0 instead of TF0.12.1, and fix this problem. But I still don't know what happen in TF0.12.1.

Cassandra 2.0 cluster's PasswordAuthenticator fails to init default user: "some nodes were not ready"

How to setup this? I have a three node cluster with replication factor 3, and getting the following errors at startup:
WARN [NonPeriodicTasks:1] 2013-10-08 14:06:42,751 PasswordAuthenticator.java (line 242) PasswordAuthenticator skipped default user setup: some nodes were not ready
WARN [NonPeriodicTasks:1] 2013-10-08 14:06:42,763 Auth.java (line 223) Skipped default superuser setup: some nodes were not ready
No default user is created, and trying to log in with e.g. cqlsh fails:
$ cqlsh mynode myport -u cassandra -p cassandra
Traceback (most recent call last):
File "/usr/opt/apache-cassandra-2.0.1/bin/cqlsh", line 2031, in <module>
main(*read_options(sys.argv[1:], os.environ))
File "/usr/opt/apache-cassandra-2.0.1/bin/cqlsh", line 2017, in main
display_float_precision=options.float_precision)
File "/usr/opt/apache-cassandra-2.0.1/bin/cqlsh", line 477, in __init__
cql_version=cqlver, transport=transport)
File "/usr/opt/apache-cassandra-2.0.1/bin/../lib/cql-internal-only-1.4.0.zip/cql-1.4.0/cql/connection.py", line 143, in connect
File "/usr/opt/apache-cassandra-2.0.1/bin/../lib/cql-internal-only-1.4.0.zip/cql-1.4.0/cql/connection.py", line 59, in __init__
File "/usr/opt/apache-cassandra-2.0.1/bin/../lib/cql-internal-only-1.4.0.zip/cql-1.4.0/cql/thrifteries.py", line 157, in establish_connection
File "/usr/opt/apache-cassandra-2.0.1/bin/../lib/cql-internal-only-1.4.0.zip/cql-1.4.0/cql/cassandra/Cassandra.py", line 455, in login
File "/usr/opt/apache-cassandra-2.0.1/bin/../lib/cql-internal-only-1.4.0.zip/cql-1.4.0/cql/cassandra/Cassandra.py", line 476, in recv_login
cql.cassandra.ttypes.AuthenticationException: AuthenticationException(why='Username and/or password are incorrect')
The error message "some nodes were not ready" is somewhat misguiding, as it seems that the reason was having two other nodes in the cluster up and running but configured to still use the AllowAllAuthenticator. After bringing down the other nodes and starting up the one with PasswordAuthentitor, creating default user succeeds:
INFO [NonPeriodicTasks:1] 2013-10-09 14:44:57,028 PasswordAuthenticator.java (line 237) PasswordAuthenticator created default user 'cassandra'
INFO [NonPeriodicTasks:1] 2013-10-09 14:44:57,760 Auth.java (line 218) Created default superuser 'cassandra'
In my case it was because the datacenter name was changed and then rolled out to the config files (cassandra-topology.properties) thanks to Puppet. Therefore the already configured NetworkTopologyStrategy for system_auth failed.
You have to change "authenticator:" and "authorizer:" in serial not in parallel when starting the node first time with these options. This way worked for me when struggling with above error.
You should also check conf/cassandra-topology.properties if you are using PropertyFileSnitch. It should be same across all the nodes

Twisted errors in Scrapy spider

When I run the spider from the Scrapy tutorial I get these error messages:
File "C:\Python26\lib\site-packages\twisted\internet\base.py", line 374, in fireEvent DeferredList(beforeResults).addCallback(self._continueFiring)
File "C:\Python26\lib\site-packages\twisted\internet\defer.py", line 195, in addCallback callbackKeywords=kw)
File "C:\Python26\lib\site-packages\twisted\internet\defer.py", line 186, in addCallbacks self._runCallbacks()
File "C:\Python26\lib\site-packages\twisted\internet\defer.py", line 328, in_runCallbacks self.result = callback(self.result, *args, **kw)
--- <exception caught here> ---
File "C:\Python26\lib\site-packages\twisted\internet\base.py", line 387, in _continueFiring callable(*args, **kwargs)
File "C:\Python26\lib\site-packages\twisted\internet\posixbase.py", line 356, in listenTCP p.startListening()
File "C:\Python26\lib\site-packages\twisted\internet\tcp.py", line 858, in startListening raise CannotListenError, (self.interface, self.port, le) twisted.internet.error.CannotListenError: Couldn't listen on any:6023: [Errno 10048]
Only one usage of each socket address (protocol/network address/port) is normally permitted.
Does anyone know what they are and how I can get rid of them?
Thanks
Perhaps you're running two Scrapy process simultaneously with telnet console enabled?.
If you want to run more than one Scrapy process at the same time, you must disable (or, at least, change the port) of web and telnet consoles.