I am running scrapyd 1.2 with scrapy version 2.1 and suddenly the daemon stoped working properly. It will schedule jobs, but they end after one second to status "finished" and the log file of this spider shows this as the last line:
2020-05-17 07:28:56 [scrapy.extensions.telnet] INFO: Telnet Password: d65ea0bd25f75ea0
There is no exit message or error message why it stopped or failed.
I installed spidermon yesterday among with other things, but on the development server everything runs OK and the production server successfully finished another job, started several hours ago. Just new spiders will fail now.
In the meantime I stoped the daemon and started it again which now outputs logs directly in the terminal.
Scheduling a spider returns:
File "<frozen importlib._bootstrap>", line 973, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'spidermon'
So clearly the install did not work as expected.
Now it boils down to two questions:
How to configure scrapyd to write a sys logfile to /var/logs/scrapyd/
How to install spidermon
Fixed the issue by uninstalling spidermon and reinstalling system wide:
sudo pip3 uninstall spidermon
sudo -H pip3 install spidermon
The logfile problematic is unresolved but to be fair not part of this question.
Related
everyone i'm using oracle arm/amd processor servers with ubuntu 22.04
i have searching for an solution for this over 1 week more than 100 hours
but i had no luck to figure out what is the problem
so if you know about this please help me!
what is problem :
here's a simply code with pyperclip
import pyperclip
string = 'testing'
pyperclip.copy(string)
a = pyperclip.paste()
print()
whenever i execute this code directly in the cli , it is okay to run
However whenever i run this code with schedulers like Crontab or Systemd.timer
(xclip && xsel is already installed)
i got this error and cannot run the code like below
Traceback (most recent call last):
File "/home/ubuntu/web_selenium/pytest.py", line 10, in <module>
pyperclip.copy(string)
File "/usr/lib/python3/dist-packages/pyperclip/__init__.py", line 689, in lazy_load_stub_copy
return copy(text)
File "/usr/lib/python3/dist-packages/pyperclip/__init__.py", line 358, in __call__
raise PyperclipException(EXCEPT_MSG)
pyperclip.PyperclipException:
Pyperclip could not find a copy/paste mechanism for your system.
For more information, please visit https://pyperclip.readthedocs.io/en/latest/index.html#not-implemented-error
i tried most of solutions in stackoverflow, mostly it was about setting path for interpreter and packages
of course i installed xclip and xsel as well / everything is good when i run code directly through cli
but only when i run code or shell script through Crontab , Systemd.timer, this error is generated
what i have tried
1. run pyperclip code through shell script with various python env PATH
i tried to lay all the possible env path like /usr/bin:/usr/local/bin .......
and even my python pakages path as .profile and ~./bashrc - failed
2. set new venv and install all the modules, run code - failed
3. run crontab as root - failed
4. install new ubuntu and switch server ubuntu amd -> ubuntu aarch64 - failed
5. reinstall module - failed
6. set shebang inside of python script ( whereis python3 - inside of python file - #!/usr/bin/python3.10) - failed
7. set shell into crontab (SHELL=/bin/bash) - failed
now i'm thinking this is the problem of pyperclip
pleaese help me!
can someone help me solve/remove these 2 problems in my Mac terminal? I cannot understand why they are appearing and how to fix them. They are appearing every time I open a new terminal window!
I have installed php, node and composer using brew and an ssh key for gitlab.
I need a step by step procedure to uninstall RUNDECK. i am facing some STACK overflow issue which i wasn't able to resolve so i want to uninstall and install it from scratch
Stack error:
[2020-06-05 18:48:44.098] ERROR StackTrace --- [tp1284944245-71] Full Stack Trace:
org.grails.taglib.GrailsTagException: [views/layouts/base.gsp:184] Error executing tag <g:render>: [views/common/_sidebar.gsp:128] Error executing tag <g:ifMenuItems>: Method 'java.util.Set com.dtolabs.rundeck.core.authorization.providers.EnvironmentalContext.forProject(java.lang.String)' must be InterfaceMethodref constant
at org.grails.gsp.GroovyPage.throwRootCause(GroovyPage.java:473)
at org.grails.gsp.GroovyPage.invokeTag(GroovyPage.java:415)
at jdk.internal.reflect.GeneratedMethodAccessor217.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:98)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.in```
WAR based instance:
Make sure that the Rundeck process is down, identify the process
doing ps aux| grep -i rundeck and use kill -9 <PID> to shut
down.
Wipe the instance, you can delete all directory (and content)
defined in %RDECK_BASE%. All configurations and files are inside
this directory. If your system has a init script to launch rundeck, ensure that script doesn't have any reference to rundeck.
Re-install following this.
RPM-based (CentOS, RHEL, Fedora) instance:
Shutdown the Rundeck service: # systemctl stop rundeckd.
Make sure that the process is down: # systemctl status rundeckd.
Remove the package, do # yum remove rundeck.
Some files keep on the system, check and wipe the following paths:
/etc/rundeck, /var/lib/rundeck and /var/log/rundeck.
Re-install following this.
DEB-based (Debian, Ubuntu, Mint) instance:
Shutdown the Rundeck service: # systemctl stop rundeckd.
Make sure that the process is down doing # systemctl status rundeckd.
Remove the package, do # apt-get purge rundeck
Some files keep on the system, check and wipe the following paths:
/etc/rundeck, /var/lib/rundeck and /var/log/rundeck.
Reinstall following this.
Anyway, I recommend to make a backup of your instance / configurations before wiping it.
For testing the best option is to run Rundeck docker image, it saves a lot of time.
About the error, check your Rundeck version, maybe you're facing this issue.
I'm trying to get unison working after upgrading to Mac OS X Catalina. Unfortunately, macports installs a more recent version of ocaml (4.08.1), which means that the unison 2.51.2 release won't compile.
Well, that's no problem, I just update to git master on unison, and recompile. Unfortunately, this fails at sync time because the version of ocaml used to compile on the mac (4.08.1) is different from the one used to compile on the other machine (4.07.1). Sigh. Okay, use opam magic to install 4.07.1 on my machine. Everything should be fine, right? No!
Here's the error:
Connected [//zzzmyhost//home/clements/unison-home -> //zzzmyotherhost//Users/clements/clements]
Looking for changes
Uncaught exception Failure("input_value: ill-formed message")
Raised at file "/private/tmp/unison/src/lwt/lwt.ml", line 126, characters 16-23
Called from file "/private/tmp/unison/src/lwt/generic/lwt_unix_impl.ml", line 102, characters 8-23
Called from file "/private/tmp/unison/src/update.ml" (inlined), line 2105, characters 2-69
Called from file "/private/tmp/unison/src/uitext.ml", line 978, characters 16-56
Called from file "/private/tmp/unison/src/uitext.ml", line 1066, characters 6-90
Called from file "/private/tmp/unison/src/uitext.ml", line 1088, characters 19-66
Called from file "/private/tmp/unison/src/uitext.ml", line 1144, characters 21-43
What's going on?
Sigh... the problem here (very non-obvious) is actually with a corrupted/wrong-format syncronization file, created when doing the failed sync in the earlier test.
The solution is just to go into ~/Library/Application Support/Unison (on a UNIX machine this path would presumably live in ~/.unison and delete the archive file that's causing the problem (probably the most recent one). In a pinch, just delete all of the archive files and start over.
I've got the same problem between Windows and Ubuntu 20.04 after upgrading from Ubuntu 18.04. I tried the binary from Ubuntu 18.04 in 20.04, which still fails, so the incompatibility is likely inside one of the dependencies.
As a workaround I created a Docker image based on Ubuntu 18.04:
FROM ubuntu:18.04
RUN apt-get update && apt-get dist-upgrade -y
RUN apt-get install unison -y
RUN useradd martin --home /home/martin
WORKDIR /home/martin
USER martin
Building it with docker build -t unison:18.04 .
And then I added a wrapper to ~/bin/unison-2.48.4-docker:
#!/bin/bash
docker run --rm -i \
-v /home/martin/dirtosync:/home/martin/dirtosync \
-v /home/martin/.unison:/home/martin/.unison \
--hostname $(hostname) \
unison:18.04 unison "$#"
Setting the --hostname is important, since the hostname is part of the archive file.
Inside the profile on my Windows machine I configured:
servercmd = ~/bin/unison-2.48.4-docker
In my setup with two windows clients and one Ubuntu 18.04 server, connected by ssh, the problem startet with a second server running on Ubuntu 20.04. Neither the old server nor the windows clients could sync with the new machine.
My solution: Copying the binary from Ubuntu 18.04 to a new directory in the Ubuntu 20.04 machine. This new file is referenced in the "authorized_keys" file of ssh on the new machine.
So far, everything works great with unison 2.48.4.
I have scrapy and scrapyd installed on a debian machine. I log in to this server using a ssh-tunnel. I then start scrapyd by going:
scrapyd
Scrapyd starts up fine and I then open up another ssh-tunnel to the server and schedule my spider with:
curl localhost:6800/schedule.json -d project=myproject -d spider=myspider
The spider runs nicely and everything is fine.
The problem is that scrapyd stops running when I quit the session where I started up scrapyd. This prevents me from using cron to schdedule spiders with scrapyd since scrapyd isn't running when the cronjob is launched.
My simple question is: How do I keep scrapyd running so that it doesn't shut down when I quit the ssh session.
Run it in a screen session:
$ screen
$ scrapyd
# hit ctrl-a, then d to detach from that screen
$ screen -r # to re-attach to your scrapyd process
You might consider launching scrapyd with supervisor.
And there is a good .conf script available as a gist here:
https://github.com/JallyHe/scrapyd/blob/master/supervisord.conf
How about ?
$ sudo service scrapyd start