I get a weird error in my log-files when I start a Zope instance. The instance is running with a ZEO server and the Zope installation is a virtualenv (in /home/myUser/opt). I get this error with several Products, but Zope is working fine and these products are installed. Here is an example with the product BTreeFolder2:
2014-01-22T12:38:13 ERROR Application Couldn't install BTreeFolder2
Traceback (most recent call last):
File "/home/myUser/opt/Zope2-2.13.21/local/lib/python2.7/site-packages/Zope2-2.13.21-py2.7.egg/OFS/Application.py", line 693, in install_product
transaction.commit()
File "/home/myUser/opt/Zope2-2.13.21/local/lib/python2.7/site-packages/transaction-1.1.1-py2.7.egg/transaction/_manager.py", line 89, in commit
return self.get().commit()
File "/home/myUser/opt/Zope2-2.13.21/local/lib/python2.7/site-packages/transaction-1.1.1-py2.7.egg/transaction/_transaction.py", line 329, in commit
self._commitResources()
File "/home/myUser/opt/Zope2-2.13.21/local/lib/python2.7/site-packages/transaction-1.1.1-py2.7.egg/transaction/_transaction.py", line 446, in _commitResources
rm.tpc_vote(self)
File "/home/myUser/opt/Zope2-2.13.21/local/lib/python2.7/site-packages/ZODB3-3.10.5-py2.7-linux-x86_64.egg/ZODB/Connection.py", line 781, in tpc_vote
s = vote(transaction)
File "/home/myUser/opt/Zope2-2.13.21/local/lib/python2.7/site-packages/ZODB3-3.10.5-py2.7-linux-x86_64.egg/ZEO/ClientStorage.py", line 1098, in tpc_vote
return self._check_serials()
File "/home/myUser/opt/Zope2-2.13.21/local/lib/python2.7/site-packages/ZODB3-3.10.5-py2.7-linux-x86_64.egg/ZEO/ClientStorage.py", line 929, in _check_serials
raise s
ConflictError: database conflict error (oid 0x01, class OFS.Application.Application, serial this txn started with 0x03a449da3a7b1e44 2014-01-22 11:38:13.706468, serial currently committed 0x03a449da3af74dee 2014-01-22 11:38:13.820164)
I'd like to fix this, even if it does not affect the functionality of my site, but I don't know where to look. Any suggestions? :)
Or does this just mean that the cached objects have to be renewed with the data of the ZEO server?
When a Zope instances starts up, it registers extensions (Products) in the ZODB. When you run multiple instances sharing a ZEO server, however, and they all start up roughly the same time, you run into conflicts during this stage. The conflicts are essentially harmless, they only occur because another instance did succeed in installing the persistent components.
The solution is to configure instances not to register products, except for one (usually one per machine in a multi-machine cluster); then on restart, only one of the instances does the registration, the rest, running the same software stack, won't have to.
Note that in more recent Zope installations, the persistent parts for Products have been deprecated and mostly disabled; if you don't use Through-The-Web products or ZClasses, you probably don't need to enable this at all.
When you do need it, set the enable-product-installation configuration in zope.conf to on for just one instance, off for the rest. If you use buildout, and the plone.recipe.zope2instance recipe, you can specify this setting in the buildout recipe configuration.
Related
I have a question regarding an interesting error I'm getting from a Python (3) program being started by systemd. This is all happening on a Raspberry Pi Zero running a fully updated Raspberry Pi OS. It's the brain of a Google AIY Voice Kit v2, though that doesn't seem to be terribly important here.
The systemd service in question runs my Python program, which calls aiy.voice.tts.say("Example text"). However, this returns a FileNotFoundError - the full traceback is:
May 28 21:50:11 voicekit-zero autostart.sh[620]: Traceback (most recent call last):
May 28 21:50:11 voicekit-zero autostart.sh[620]: File "/home/pi/ready.py", line 27, in <module>
May 28 21:50:11 voicekit-zero autostart.sh[620]: """, volume=5)
May 28 21:50:11 voicekit-zero autostart.sh[620]: File "/home/pi/AIY-projects-python/src/aiy/voice/tts.py", line 52, in say
May 28 21:50:11 voicekit-zero autostart.sh[620]: with tempfile.NamedTemporaryFile(suffix='.wav', dir=RUN_DIR) as f:
May 28 21:50:11 voicekit-zero autostart.sh[620]: File "/usr/lib/python3.7/tempfile.py", line 686, in NamedTemporaryFile
May 28 21:50:11 voicekit-zero autostart.sh[620]: (fd, name) = _mkstemp_inner(dir, prefix, suffix, flags, output_type)
May 28 21:50:11 voicekit-zero autostart.sh[620]: File "/usr/lib/python3.7/tempfile.py", line 397, in _mkstemp_inner
May 28 21:50:11 voicekit-zero autostart.sh[620]: fd = _os.open(file, flags, 0o600)
May 28 21:50:11 voicekit-zero autostart.sh[620]: FileNotFoundError: [Errno 2] No such file or directory: '/run/user/1000/tmponnl3w02.wav'
It's reasonably clear from this traceback that the TTS script writes a WAV file to a temporary location in /run/user/1000/, then uses it for playback. By that point, however, the file has become inaccessible. My best guess is that the filesystem isn't fully initialized yet. (I'm not certain of this, and I don't have all that much experience with systemd services, so I could definitely be wrong.)
The systemd service file specifies Wants and After for both network-online.target and systemd-timesyncd.service, though of course neither of those are directly related to filesystem readiness. Is there another service I can start after that will ensure the file system is ready for this call? If not, I can just wait a few seconds, though I'd prefer to build a more robust system that should work reliably.
Thanks!
This issue turned out to be easier than I'd anticipated, once I'd filled in some missing details.
First, this wasn't a problem with Python, or a delay in mounting the filesystem - it centered on the relevant user's (in this case, pi, which has UID 1000) login state at runtime. I discovered that if I logged in via SSH, the error in question disappeared.
Researching that led me to loginctl enable-linger:
Enable/disable user lingering for one or more users. If enabled for a specific user, a user manager is spawned for the user at boot and kept around after logouts. This allows users who are not logged in to run long-running services. Takes one or more user names or numeric UIDs as argument. If no argument is specified, enables/disables lingering for the user of the session of the caller.
Running this command is all that was required to solve the issue.
I understand this is a little vague but not sure where else to go to or things to debug. My python script was running fine yesterday. I made minor changes today and now it only runs successfully on my Amazon LightSail (ec2) machine. Everything I read about segmentation errors is that there is not enough memory, however my local machine has 16gb of ram while the cloud machine only has 1gb. Plus I am not working with big files? The files being imported/manipulated are typically under 2mb and there are like 7-10 files.
I feel it may be something related to my terminal/zsh rather than my codes.
The below is the error code I can not seem to manage to get around.
I've done enough research to find the python faulthandler module import faulthandler; faulthandler.enable() to give the debugging below:
Fatal Python error: Segmentation fault
Current thread 0x000000010c58edc0 (most recent call first):
File "/Users/garrett/opt/anaconda3/lib/python3.7/site-packages/pandas/core/groupby/generic.py", line 1795 in <genexpr>
File "/Users/garrett/opt/anaconda3/lib/python3.7/site-packages/pandas/core/groupby/generic.py", line 1797 in <listcomp>
File "/Users/garrett/opt/anaconda3/lib/python3.7/site-packages/pandas/core/groupby/generic.py", line 1797 in count
File "GmailDownloader.py", line 215 in <module>
zsh: segmentation fault python *.py
The code seems to regularly break on line 215 while trying to compute a gorupby in pandas but it is very similar to other groupbys in the code that were successful before it.
I am on a Mac Catlina using the pre-baked zsh for my terminal handling but even when I switch to good ol' bash using chsh -s /bin/bash in my terminal and then running the code I still get a zsh segmentation error.
I have recently tried out PyCharm today and it asked for permissions to store something in a bin folder to which I just said yes. I'm not sure if that is correlated at all or not.
The full code repository: https://github.com/GarrettMarkScott/AutomotiveCRMPuller
Ongoing list of other things I have tried:
Trashing the Terminal preferences (~/Library/Preferences/com.apple.Terminal.plist)
I almost threw in the towel but tried to reinstall my pandas since it was mentioned in my bug error and what do you know it worked after running pip install --upgrade pandas
Would of been impossible without the FaultHandler! Hopefully this helps someone out there!
This is driving me nuts.
I'm setting up airflow in a cloud environment. I have one server running the scheduler and the webserver and one server as a celery worker, and I'm using airflow 1.8.0.
Running jobs works fine. What refuses to work is logging.
I've set up the correct path in airflow.cfg on both servers:
remote_base_log_folder = s3://my-bucket/airflow_logs/
remote_log_conn_id = s3_logging_conn
I've set up s3_logging_conn in the airflow UI, with the access key and the secret key as described here.
I checked the connection using
s3 = airflow.hooks.S3Hook('s3_logging_conn')
s3.load_string('test','test',bucket_name='my-bucket')
This works on both servers. So the connection is properly set up. Yet all I get whenever I run a task is
*** Log file isn't local.
*** Fetching here: http://*******
*** Failed to fetch log file from worker.
*** Reading remote logs...
Could not read logs from s3://my-bucket/airflow_logs/my-dag/my-task/2018-02-15T21:46:47.577537
I tried manually uploading the log following the expected conventions and the webserver still can't pick it up - so the problem is on both ends. I'm at a loss at what to do, everything I've read so far tells me this should be working. I'm close to just installing the 1.9.0 which I hear changes logging and see if I'm more lucky.
UPDATE: I made a clean install of Airflow 1.9 and followed the specific instructions here.
Webserver won't even start now with the following error:
airflow.exceptions.AirflowConfigException: section/key [core/remote_logging] not found in config
There is an explicit reference to this section in this config template.
So I tried removing it and just loading the S3 handler without checking first and I got the following error message instead:
Unable to load the config, contains a configuration error.
Traceback (most recent call last):
File "/usr/lib64/python3.6/logging/config.py", line 384, in resolve:
self.importer(used)
ModuleNotFoundError: No module named
'airflow.utils.log.logging_mixin.RedirectStdHandler';
'airflow.utils.log.logging_mixin' is not a package
I get the feeling that this shouldn't be this hard.
Any help would be much appreciated, cheers
Solved:
upgraded to 1.9
ran the steps described in this comment
added
[core]
remote_logging = True
to airflow.cfg
ran
pip install --upgrade airflow[log]
Everything's working fine now.
I'm using WebLogic 10.3.6 (it's old, I know) with Java6 and am trying to install in a new environment. I'm using WLST scripts that worked in other environments where the OS is the same (CentOS 5.11), SELinux is in permissive mode, and the application user has permission to write to the directory where all of the WLS stuff is saved. Each time I try to update anything in the SecurityConfiguration MBean, it get a timeout when trying to activate. Initially I thought it was just for the node manager credentials, but I tried to do
!> cmo.setClearTextCredentialAccessEnabled(true)
!> validate()
!> save()
!> activate()
Everything is fine until the activate... here's my results:
Activating all your changes, this may take a while ...
The edit lock associated with this edit session is released
once the activation is completed.
Traceback (innermost last):
File "<console>", line 1, in ?
File "<iostream>", line 376, in activate
File "<iostream>", line 1847, in raiseWLSTException
WLSTException: Error occured while performing activate : Error while Activating changes. : [DeploymentService:290053]Request with id '1,488,512,657,383' timed out on admin server.
Use dumpStack() to view the full stacktrace
What am I missing?
The real issue causing this problem was the NFS where my binaries were being shared across instances. The NFS was configured in a VERY non-standard way, causing all kinds of file read/write timeouts.
We have an auto-deployment script that uses wsadmin and jython. The script appears to work as expected however after 6-7 redeployments the AdminTask object becomes unavilable, resulting in the following error when we attempt to use that object:
WASX7209I: Connected to process "server1" on node ukdlniqa41Node01 using SOAP connector; The type of process is: UnManagedProcess
WASX8011W: AdminTask object is not available.
...
Traceback (innermost last):
File "<string>", line 251, in ?
File "<string>", line 14, in main
File "<string>", line 38, in initialize
NameError: AdminTask
My question is, what would cause this AdminTask object to become unavailable? (it remains unavailable until we restart the server instance)
AdminTask may be available if one of the previous tasks does not finish properly. This happens often especially if your server is in development mode. I would suggest gathering Deployment Mustgather as per http://www-01.ibm.com/support/docview.wss?rs=180&context=SSCR4XA&q1=MustGatherDocument&uid=swg21199344&loc=en_US&cs=utf-8&lang=en and submitting the results to IBM.