I am running my spiders into celery worker. Spider scrape a website and then bunch of follow-up links. after some time spider stop processing any further.
lsof output shows that for thread , connection are in CLOSE_WAIT state
# lsof -i -n
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
celery 10 root 32u IPv4 105621511 0t0 TCP 127.0.0.1:6023 (LISTEN)
celery 10 root 33u IPv4 105603949 0t0 TCP 10.1.195.250:38162->104.17.38.150:http (ESTABLISHED)
celery 10 root 34u IPv4 105610494 0t0 TCP 10.1.195.250:41864->185.230.61.195:https (CLOSE_WAIT)
celery 10 root 35u IPv4 105614120 0t0 TCP 10.1.195.250:39742->185.230.61.195:http (CLOSE_WAIT)
celery 10 root 36u IPv4 105603950 0t0 TCP 10.1.195.250:52672->185.230.61.96:http (CLOSE_WAIT)
celery 10 root 37u IPv4 105620542 0t0 TCP 10.1.195.250:38200->209.236.228.178:http (CLOSE_WAIT)
celery 10 root 38u IPv4 105603948 0t0 TCP 10.1.195.250:51848->35.208.181.87:http (CLOSE_WAIT)
celery 10 root 39u IPv4 105614124 0t0 TCP 10.1.195.250:56290->185.230.61.96:https (CLOSE_WAIT)
celery 10 root 40u IPv4 105604983 0t0 TCP 10.1.195.250:43118->216.185.90.112:http (CLOSE_WAIT)
celery 10 root 41u IPv4 105618465 0t0 TCP 10.1.195.250:55006->209.59.212.167:http (CLOSE_WAIT)
celery 10 root 45u IPv4 105600888 0t0 TCP 10.1.195.250:34572->23.227.38.74:http (ESTABLISHED)
celery 10 root 46u IPv4 105620539 0t0 TCP 10.1.195.250:35846->205.178.189.129:http (CLOSE_WAIT)
celery 10 root 48u IPv4 105620541 0t0 TCP 10.1.195.250:39674->185.230.61.195:http (CLOSE_WAIT)
celery 10 root 49u IPv4 105610495 0t0 TCP 10.1.195.250:49450->178.128.150.108:http (CLOSE_WAIT)
celery 10 root 51u IPv4 105614122 0t0 TCP 10.1.195.250:53770->23.227.38.74:https (ESTABLISHED)
celery 10 root 52u IPv4 105614123 0t0 TCP 10.1.195.250:52930->54.86.91.237:https (CLOSE_WAIT)
celery 10 root 53u IPv4 105614125 0t0 TCP 10.1.195.250:37998->209.236.228.178:https (CLOSE_WAIT)
celery 10 root 54u IPv4 105614126 0t0 TCP 10.1.195.250:59992->35.208.181.87:https (CLOSE_WAIT)
celery 10 root 55u IPv4 105605002 0t0 TCP 10.1.195.250:39692->192.124.249.18:http (CLOSE_WAIT)
celery 10 root 56u IPv4 105612653 0t0 TCP 10.1.195.250:41912->185.230.61.195:https (CLOSE_WAIT)
celery 10 root 57u IPv4 105612657 0t0 TCP 10.1.195.250:47560->104.197.82.118:http (CLOSE_WAIT)
celery 10 root 58u IPv4 105612656 0t0 TCP 10.1.195.250:33926->209.59.212.167:https (CLOSE_WAIT)
celery 10 root 59u IPv4 105614129 0t0 TCP 10.1.195.250:41614->178.128.150.108:https (CLOSE_WAIT)
celery 10 root 62u IPv4 105614131 0t0 TCP 10.1.195.250:37534->34.66.87.174:http (CLOSE_WAIT)
celery 10 root 63u IPv4 105600910 0t0 TCP 10.1.195.250:47682->166.62.115.136:https (CLOSE_WAIT)
celery 10 root 64u IPv4 105614141 0t0 TCP 10.1.195.250:43222->216.185.90.112:http (CLOSE_WAIT)
celery 10 root 65u IPv4 105600912 0t0 TCP 10.1.195.250:41060->50.63.7.227:http (CLOSE_WAIT)
celery 10 root 66u IPv4 105600913 0t0 TCP 10.1.195.250:41254->104.197.82.118:https (CLOSE_WAIT)
celery 10 root 69u IPv4 105614695 0t0 TCP 10.1.195.250:42766->104.112.162.8:https (ESTABLISHED
ps -aux shows that thread is in sleep and waiting for an event
# ps -aux
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.1 0.0 80024 62700 ? Ss 17:23 0:05 /usr/local/bin/python /usr/local/bin/celery -A data_ex
root 8 0.0 0.0 118892 76360 ? S 17:23 0:00 /usr/local/bin/python /usr/local/bin/celery -A data_ex
root 10 0.0 0.0 902592 100916 ? Sl 17:23 0:01 /usr/local/bin/python /usr/local/bin/celery -A data_ex
root 485 0.0 0.0 121900 79376 ? S 18:07 0:00 /usr/local/bin/python /usr/local/bin/celery -A data_ex
root 486 10.0 0.1 950312 144056 ? Sl 18:07 1:19 /usr/local/bin/python /usr/local/bin/celery -A data_ex
root 501 0.4 0.0 455868 62432 ? Sl 18:11 0:02 /usr/local/bin/python /usr/local/bin/celery flower -A
root 508 0.3 0.0 121916 79388 ? S 18:17 0:00 /usr/local/bin/python /usr/local/bin/celery -A data_ex
root 509 22.4 0.1 958724 154876 ? Sl 18:17 0:42 /usr/local/bin/python /usr/local/bin/celery -A data_ex
root 520 0.5 0.0 2388 700 pts/0 Ss 18:20 0:00 /bin/sh
root 526 0.0 0.0 9392 3048 pts/0 R+ 18:20 0:00 ps -aux
Starce shows that thread is waiting on fd 69
# strace -p 10
strace: Process 10 attached
read(69,
Seems like spider are not closing connection properly.
how do i solve this?
I thought of adding timeouts to celery task , but all thread will hit HARD_LIMIT eventually.
how to make sure that scrapy is closing each connection propery?
This most likely has to do with the code that you are using for spidering. You may have to set a timeout on the library that you are using to make your http / https requests.
Related
I'm having a redis connection established and I've a python script which performs CLUSTER MEET, CLUSTER FORGET, IPTBL ADD and IPTBL DELETE.
When I give CLUSTER FORGET.
Before FIN packet reaches the other end, it closes the connection and I'm left with a stale socket.
On going through the redis documentation and other blogs I found tcp-keepalive and even after changing it to 300 and restarting the redisd I still see no effect.
/usr/sbin/lsof | grep 16381
redis-ser 23104 admin 10u IPv6 397237 0t0 TCP *:16381 (LISTEN)
redis-ser 23104 admin 11u IPv4 397239 0t0 TCP *:16381 (LISTEN)
redis-ser 23104 admin 12u IPv4 397251 0t0 TCP a.b.c.d:16381->a.b.c.e:56440 (ESTABLISHED)
redis-ser 23104 admin 13u IPv4 397253 0t0 TCP a.b.c.d:16381->a.b.c.f:58744 (ESTABLISHED)
redis-ser 23104 admin 14u IPv4 408062 0t0 TCP a.b.c.d:47230->a.b.c.e:16381 (ESTABLISHED)
redis-ser 23104 admin 15u IPv4 748468 0t0 TCP a.b.c.d:39754->a.b.c.f:16381 (ESTABLISHED)
redis-ser 23104 admin 16u IPv4 719475 0t0 TCP a.b.c.d:16381->a.b.c.f:58868 (ESTABLISHED)
I would like to understand how to make tcp-keepalive work ? Am I missing something ?
Any help would be appreciated.
I am using a Mac Book Pro
I need to use my localhost Port :80 but i keep getting this ERROR message:
Error response from daemon: driver failed programming external
connectivity on endpoint nostalgic_ride : Error starting userland
proxy:
Bind for 0.0.0.0:80: unexpected error (Failure EADDRINUSE).
this issue arose because i tried to use docker to do this:
docker run -it -p 80:80 -p 443:443 nginx
I did a search to see what is listening on all my ports:
sudo lsof -nP | grep LISTEN
UserEvent 41 root 113u IPv6 0xa09c168d9d24ed2f 0t0 TCP [fe80:8::aede:48ff:fe00:1122]:49153 (LISTEN)
UserEvent 41 root 114u IPv6 0xa09c168d9d24e1af 0t0 TCP [fe80:8::aede:48ff:fe00:1122]:49154 (LISTEN)
UserEvent 41 root 115u IPv6 0xa09c168d9d24dbef 0t0 TCP [fe80:8::aede:48ff:fe00:1122]:49155 (LISTEN)
UserEvent 41 root 116u IPv6 0xa09c168d9d24f2ef 0t0 TCP [fe80:8::aede:48ff:fe00:1122]:49156 (LISTEN)
php-fpm 48 root 8u IPv4 0xa09c168d9d7b322f 0t0 TCP 127.0.0.1:9000 (LISTEN)
php-fpm 221 _www 9u IPv4 0xa09c168d9d7b322f 0t0 TCP 127.0.0.1:9000 (LISTEN)
php-fpm 222 _www 9u IPv4 0xa09c168d9d7b322f 0t0 TCP 127.0.0.1:9000 (LISTEN)
EEventMan 351 tom 4u IPv4 0xa09c168da7672eaf 0t0 TCP *:2968 (LISTEN)
java 648 tom 69u IPv6 0xa09c168d9d24c4ef 0t0 TCP [fe80:1::1]:49185 (LISTEN)
LogiMgrDa 715 tom 4u IPv4 0xa09c168da914f52f 0t0 TCP *:59866 (LISTEN)
httpd 860 root 4u IPv6 0xa09c168da64892ef 0t0 TCP *:80 (LISTEN)
httpd 864 _www 4u IPv6 0xa09c168da64892ef 0t0 TCP *:80 (LISTEN)
httpd 955 _www 4u IPv6 0xa09c168da64892ef 0t0 TCP *:80 (LISTEN)
Questions:
It appears that there are 3 instances of httpd listening on Port 80. If so, does anyone know what these httpd are for and whether it's safe to remove them.
If so, how do I kill/remove them.
ANSWER TO SIMILAR QUESTIONS ON THIS ISSUE:
these answers do not help me beucase they all recommend simply changing the port. however i need port 80 for localhost development
Rabbitmq keeps dying within 2 minutes of startup, and fails with the error:
ERROR: epmd error for host localhost: timeout (timed out)
The /etc/hosts file shows:
127.0.0.1 localhost []-MacBook-Pro []-MacBook-Pro.local
255.255.255.255 broadcasthost
::1 localhost
([] replaces actual name in file)
I have tried uninstalling and reinstalling Erlang (v20.3.8.3) and Rabbitmq via homebrew, and still no connection. This is also regardless of what wifi I am on, or if I'm disconnected from wifi. Other commands I have tried:
$ echo $HOSTNAME
localhost
$ hostname
localhost
$ ifconfig lo0
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384
options=1203<RXCSUM,TXCSUM,TXSTATUS,SW_TIMESTAMP>
inet 127.0.0.1 netmask 0xff000000
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
nd6 options=201<PERFORMNUD,DAD>
$ lsof -p 15279
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
epmd 15279 user cwd DIR 1,5 960 2 /
epmd 15279 user txt REG 1,5 46756 4852991 /usr/local/Cellar/erlang#20/20.3.8.3/lib/erlang/erts-9.3.3.2/bin/epmd
epmd 15279 user txt REG 1,5 841456 2333502 /usr/lib/dyld
epmd 15279 user txt REG 1,5 1170673664 3866438 /private/var/db/dyld/dyld_shared_cache_x86_64h
epmd 15279 user 0r CHR 3,2 0t0 311 /dev/null
epmd 15279 user 1w CHR 3,2 0t0 311 /dev/null
epmd 15279 user 2w CHR 3,2 0t0 311 /dev/null
epmd 15279 user 3u IPv4 0x91f821cb489372bb 0t0 TCP *:epmd (LISTEN)
epmd 15279 user 4u IPv6 0x91f821cb3ac82ed3 0t0 TCP *:epmd (LISTEN)
epmd 15279 user 5u IPv4 0x91f821cb4a2f057b 0t0 TCP localhost:epmd->localhost:55223 (ESTABLISHED)
epmd 15279 user 6u IPv4 0x91f821cb3a64dc1b 0t0 TCP localhost:epmd->localhost:55470 (ESTABLISHED)
epmd 15279 user 7u IPv4 0x91f821cb4919995b 0t0 TCP localhost:epmd->localhost:55974 (ESTABLISHED)
I'm not sure what else to try here, and am hoping someone has an idea.
I think it likely means that the port 4369 is not accessible.
Also your hosts file is a little strange:
127.0.0.1 localhost []-MacBook-Pro []-MacBook-Pro.local
can you just make that:
127.0.0.1 localhost
After getting help on the rabbitmq list, I discovered that rabbitmq was unable to log, and that's why it was quitting.
A bit more detail to help others in this, using epmd -d to start up epmd was useful to finally determine that rabbitmq was actually getting there, so that wasn't the issue in this case. But finding the erl_crash.dump file did show the error for not being able to log, so I changed the path in /usr/local/etc/rabbitmq/rabbitmq-env.conf for the log file, and now it works.
So I've been trying for the past several hours to get my port 80 opened, so that I can access my Apache server. I'm running RHEL 6.5, and below is the configuration for my iptables.
# Generated by iptables-save v1.4.7 on Wed Jul 2 12:59:50 2014
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [9:1332]
-A INPUT -m state --state NEW -m tcp -p tcp --dport 80 -j ACCEPT
## Open 443 port i.e. HTTPS
-A INPUT -m state --state NEW -m tcp -p tcp --dport 443 -j ACCEPT
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 80 -j ACCEPT
COMMIT
# Completed on Wed Jul 2 12:59:50 2014
I've saved them and restarted, to no avail. I am using a port checker (http://www.checkmyports.net/) to check if it is open, but it isn't. Before you mark this as a duplicate, I have tried everything online. I've reconfigured my iptables multiple times, removed additional firewalls, disabled and re-enabled, and multiple other solutions, all to no avail. Any ideas on where I'm going wrong? Thanks.
Output of ps aux | grep 'httpd'
:
root 20353 0.0 0.7 175704 3668 ? Ss 12:59 0:00 /usr/sbin/httpd
apache 20355 0.0 0.4 175704 2408 ? S 12:59 0:00 /usr/sbin/httpd
apache 20356 0.0 0.4 175704 2408 ? S 12:59 0:00 /usr/sbin/httpd
apache 20357 0.0 0.4 175704 2408 ? S 12:59 0:00 /usr/sbin/httpd
apache 20358 0.0 0.4 175704 2408 ? S 12:59 0:00 /usr/sbin/httpd
apache 20359 0.0 0.4 175704 2408 ? S 12:59 0:00 /usr/sbin/httpd
apache 20360 0.0 0.4 175704 2408 ? S 12:59 0:00 /usr/sbin/httpd
apache 20361 0.0 0.4 175704 2408 ? S 12:59 0:00 /usr/sbin/httpd
apache 20362 0.0 0.4 175704 2408 ? S 12:59 0:00 /usr/sbin/httpd
root 21624 0.0 0.1 103244 856 pts/0 S+ 13:55 0:00 grep httpd
Output of netstat -tulpn:
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN 960/rpcbind
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 28361/sshd
tcp 0 0 0.0.0.0:36088 0.0.0.0:* LISTEN 978/rpc.statd
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 1108/sendmail
tcp 0 0 :::111 :::* LISTEN 960/rpcbind
tcp 0 0 :::80 :::* LISTEN 20353/httpd
tcp 0 0 :::51733 :::* LISTEN 978/rpc.statd
tcp 0 0 :::22 :::* LISTEN 28361/sshd
udp 0 0 0.0.0.0:111 0.0.0.0:* 960/rpcbind
udp 0 0 0.0.0.0:39182 0.0.0.0:* 978/rpc.statd
udp 0 0 0.0.0.0:68 0.0.0.0:* 20708/dhclient
udp 0 0 0.0.0.0:711 0.0.0.0:* 960/rpcbind
udp 0 0 0.0.0.0:730 0.0.0.0:* 978/rpc.statd
udp 0 0 :::111 :::* 960/rpcbind
udp 0 0 :::711 :::* 960/rpcbind
udp 0 0 :::35278 :::* 978/rpc.statd
Ensure there is something running on that port.
If you have port 80 open on your firewall but nothing is listening on that port (apache,http) then the port will show as closed.
Whats the output of
ps aux | grep 'httpd'
and
netstat -tulpn
You could try clearing out iptables entirely, get the web access working, and then turn it back on.
I have an iptables-clear.sh script that I run to do this.
Note this doesn't use the /etc/init.d version of iptables which you might have to turn off while you're fixing this. Just remember to turn it back on once you're done.
# Flush all tables
iptables -F
iptables -t nat -F
# Default policy to ACCEPT
iptables -P INPUT ACCEPT
iptables -P FORWARD ACCEPT
iptables -P OUTPUT ACCEPT
iptables -t nat -P PREROUTING ACCEPT
iptables -t nat -P POSTROUTING ACCEPT
iptables -t nat -P OUTPUT ACCEPT
I just updated my OSX yesterday with Mavericks and now I am getting error while using apache and Jetty.
It used to map my example.com domain directly to the localhost.
Now It seems broke on do to run my application server I have to explicitly specifying port to hit my server.(ex. example.com:8080)
While I run below command.
sudo apachectl -e debug
I will get the
[debug] mod_so.c(246): loaded module jk_module
(48)Address already in use: make_sock: could not bind to address [::]:80
(48)Address already in use: make_sock: could not bind to address 0.0.0.0:80
no listening sockets available, shutting down
I ran many diff command to kill all httpd process as well find port which is listening on this port.
ex. sudo lsof -i :80
httpd 946 root 5u IPv6 0xe0c8ad50898bcf0f 0t0 TCP *:http (LISTEN)
httpd 947 _www 5u IPv6 0xe0c8ad50898bcf0f 0t0 TCP *:http (LISTEN)
httpd 949 _www 5u IPv6 0xe0c8ad50898bcf0f 0t0 TCP *:http (LISTEN)
httpd 954 _www 5u IPv6 0xe0c8ad50898bcf0f 0t0 TCP *:http (LISTEN)
Google 1633 jignesh.sheth 59u IPv4 0xe0c8ad508597efaf 0t0 TCP example.com:53742->stackoverflow.com:http (ESTABLISHED)
Google 1633 jignesh.sheth 189u IPv4 0xe0c8ad5082027faf 0t0 TCP example.com:53725->stackoverflow.com:http (ESTABLISHED)
Any idea what am I doing wrong?
I also had the same, I removed 'Listern 80' from my config file to solve this issue.
I had the same error. The error log file for Apache server at /var/log/apache*/ mentioned "Address already in use: make_sock: could not bind to address 0.0.0.0:443".
Resolved it using "sudo lsof -i :" command to find processes running on port.
Kill all the processes and re-run sudo apachectl start.
The apache server should start running.
This may help: https://superuser.com/questions/479146/macports-apache2-could-not-bind-to-address/479147#479147?newreg=95099c6a74724da49640329d4ee400f2
I had the same problem.
What I did was changing the permission on "username.conf" from users directory to 644 like in this tutorial
http://www.coolestguidesontheplanet.com/downtown/get-apache-mysql-php-and-phpmyadmin-working-osx-109-mavericks