I have a crawler running since a few days. I want to pause the crawler in order to do something else on the system. Scrapy documentation says, this can be done using telnet console but I am unable to login into telnet console. Here are the processes running in the system:
[root#xxx tmp]# telnet localhost 6073
Trying 127.0.0.1...
telnet: connect to address 127.0.0.1: Connection refused
[root#xxx tmp]# ps aux | grep scrapy
root 5504 0.0 0.0 110400 860 pts/1 S+ 04:31 0:00 grep scrapy
root 31457 4.0 1.9 774880 299436 pts/1 Sl Sep21 141:27 /usr/local/pyenv/bin/python2.7 /usr/local/pyenv/bin/scrapy crawl myCrawler
Any help is appreciated. Thanks.
Hah, here I am answering my own question. As stated in documentation, scrapy runs on port range [6023, 6073]. So to find the port being used, I used this command:
netstat -l
Output:
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 xxx.localdomain:6025 *:* LISTEN
tcp 0 0 *:27017 *:* LISTEN
tcp 0 0 *:mysql *:* LISTEN
tcp 0 0 *:6379 *:* LISTEN
tcp 0 0 *:webcache *:* LISTEN
"6025" port is what I was looking for.
Related
After following the installation instructions referenced at https://superset.apache.org/docs/installation/installing-superset-from-scratch, I'm not able to see the app at the ip number/server port in my win 10 web browser. How do I make it work at the right ip address?
I've installed it under venv running on a CentOS 8 VM on my win 10 laptop. I'm using the NAT network adapter and I can can use putty and the CentOS 8 cockpit app is available on port 9090.
Currently it says it's running at 127.0.0.1:8089:
(venv) /root>superset run -p 8089 --with-threads --reload --debugger
logging was configured successfully
2021-08-04 15:35:48,492:INFO:superset.utils.logging_configurator:logging was configured successfully
2021-08-04 15:35:48,505:INFO:root:Configured event logger of type <class 'superset.utils.log.DBEventLogger'>
/root/venv/lib64/python3.8/site-packages/flask_caching/__init__.py:201: UserWarning: Flask-Caching: CACHE_TYPE is set to null, caching is effectively disabled.
warnings.warn(
No PIL installation found
2021-08-04 15:35:48,722:INFO:superset.utils.screenshots:No PIL installation found
* Serving Flask app "superset" (lazy loading)
* Environment: production
WARNING: This is a development server. Do not use it in a production deployment.
Use a production WSGI server instead.
* Debug mode: off
2021-08-04 15:35:50,007:INFO:werkzeug: * Running on http://127.0.0.1:8089/ (Press CTRL+C to quit)
netstat shows port 8089 open for 127.0.0.1, but not the VM's ip number. nmap shows the port closed.
/root>netstat -tlpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 944/sshd
tcp 0 0 127.0.0.1:8089 0.0.0.0:* LISTEN 4247/python3
tcp6 0 0 :::9090 :::* LISTEN 1/systemd
tcp6 0 0 :::22 :::* LISTEN 944/sshd
/root>nmap 192.168.42.130
Starting Nmap 7.70 ( https://nmap.org ) at 2021-08-04 15:45 PDT
Nmap scan report for kevinsAppServer (192.168.42.130)
Host is up (0.000015s latency).
Not shown: 998 closed ports
PORT STATE SERVICE
22/tcp open ssh
9090/tcp open zeus-admin
Nmap done: 1 IP address (1 host up) scanned in 1.73 seconds
/root>nmap -p 8089 192.168.42.130
Starting Nmap 7.70 ( https://nmap.org ) at 2021-08-04 15:45 PDT
Nmap scan report for kevinsAppServer (192.168.42.130)
Host is up (0.000067s latency).
PORT STATE SERVICE
8089/tcp closed unknown
I think it should work if i can get the port opened on my server's IP number, unless there's something I did wrong..
I got it to work by adding the -h option for the superset run command, which I found when running the superset run --help at the server command line. So now this command works:
superset run -h my.i.p.adddress -p 8089 --with-threads --reload --debugger
In my case I also had to open port 8089 on the CentOS firewall.
I have a running apache webserver in centos docker container which is hosted by AWS redhat instance.
I am able to curl my container webserver in AWS instance local host but unable to access publicly in my laptop web browser.
details of the Webserver:
docker run -d --name httpd -p 8080:8080 -v /home/ec2-user/apache/web1/www:/var/www:Z docker.io/centos/httpd-24-centos7
The output of docker ps:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
abd790b28b51 docker.io/centos/httpd-24-centos7 "container-entrypo..." 2 hours ago Up 20 minutes 0.0.0.0:8080->8080/tcp, 8443/tcp httpd
In AWS instance :
curl http://localhost:8080
Hello World!!!
But unable to get this in public internet using AWS host public ip from my laptop.
The output of netstat -tulpn:
(No info could be read for "-p": geteuid()=1000 but you should be root.)
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN -
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN -
tcp6 0 0 :::8080 :::* LISTEN -
tcp6 0 0 :::22 :::* LISTEN -
tcp6 0 0 ::1:25 :::* LISTEN -
udp 0 0 0.0.0.0:68 0.0.0.0:* -
udp 0 0 127.0.0.1:323 0.0.0.0:* -
udp6 0 0 ::1:323 :::* -
My AWS Security inbound rules:
HTTP TCP 80 157.51.138.196/32
All traffic All All 157.51.138.196/32
SSH TCP 22 157.51.138.196/32
DNS (TCP) TCP 53 157.51.138.196/32
HTTPS TCP 443 157.51.138.196/32
you are missing inbound rule 8080 on instance security group add it and check and inform further if not work.
Hurray it Worked!!!
But not sure how it worked actually ;-) Will try to find out that...
I just stopped all apache containers and removed them entirely. Then executed the below command and everything worked perfect.
While using the below make sure you are mapping the volume (-v) correctly to the index.html file in your host.
docker run -d --name httpd -p 9080:8080 -v /home/ec2-user/apache/web1/www/html:/var/www/html:Z docker.io/centos/httpd-24-centos7
Even though tunnelling using ssh is a well discussed topic in internet, I couldn't make my setup proper. This is what the problem statement I have
-> I have two servers A and B.
-> I am trying to create a tunnel from A to B
-> In B I have echo server which is listening on port 34567
-> In A I have executed the command ssh -L 4444:10.106.251.90:34567 127.0.0.1
-> Netstat shows as follows
~ # netstat -na | grep 4444
tcp 0 0 127.0.0.1:4444 0.0.0.0:* LISTEN
tcp 0 0 ::1:4444 :::* LISTEN
-> In B I tried to check netstat -na | grep 4444 but there was no reference of it
-> In A I have echo client which writes into 127.0.0.1:4444
-> If I execute client in A, it doesn't forward. Instead it gives some 0 length string always
Is there any thing that I am missing here ?
Thanks
~S
Try using
ssh -L 4444:127.0.0.1:34567 user#10.106.251.90
This will open port 4444 on your local machine, forwarding packets to port 34567 of the remote machine.
# using on local
telnet 127.0.0.1 4444
# is the same as
# on remote
telnet 127.0.0.1 34567
I have ubuntu 14.04 server edition with nodejs, mongodb and nginx installed. This worked fine until yesterday. My internet went down for about 8 hours because of a storm and upon coming back up, nodejs works, mongo works, but anyone connecting to port 80 gets ERR_CONNECTION_REFUSED. i attempted to switch the listening port just to see what would happen and got the same result. I can use the server's internal IP and i still get ERR_CONNECTION_REFUSED, however visiting port 80 via lynx on the server itself via localhost or 127.0.0.1, the application works just fine. I have also tried this using apache instead of nginx and it does not work either. ive disabled my ufw completely, restarted the server, double and triple checked configurations. even netstat says the server is listening on port 80, an nmap scan shows port 80 open and listening, but trying to connect to it gives ERR_CONNECTION_REFUSED
I do not know what to do and based on google and stack overflow search results, im the first person in the history of web servers to ever have to ask this question, so alas i could find nothing helpful anywhere.
Thanks in advance
UFW Status
user#io# ufw status
Status: inactive
Nginx Status
user#io# service nginx status
* nginx is running
Nginx Access Log
user#io:/var/log/nginx# cat access.log
::1 - - [26/Feb/2016:16:04:23 -0600] "GET / HTTP/1.0" 200 7746 "-" "Lynx/2.8.8pre.4 libwww-FM/2.14 SSL-MM/1.4.1 GNUTLS/2.12.23"
netstat
user#io# netstat -tulpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 127.0.0.1:27017 0.0.0.0:* LISTEN 977/mongod
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 23009/nginx
tcp 0 0 127.0.0.1:28017 0.0.0.0:* LISTEN 977/mongod
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 974/sshd
tcp 0 0 0.0.0.0:9561 0.0.0.0:* LISTEN 2083/node
tcp6 0 0 :::80 :::* LISTEN 23009/nginx
tcp6 0 0 :::22 :::* LISTEN 974/sshd
udp 0 0 0.0.0.0:27712 0.0.0.0:* 808/dhclient
udp 0 0 0.0.0.0:68 0.0.0.0:* 808/dhclient
udp6 0 0 :::52391 :::* 808/dhclient `
Try iptables -t nat -F to clear all pre-routing rules.
I am running a java application on a remote server that opens port 7462 for a telnet connection to issue simple commands. I have established a VPN connection (using openvpn) between my local machine and the remote server, but cannot telnet from my local machine to port 7462. The IP address of my vpn server is 10.8.0.1. I am also running postfix on my remote server and am able to telnet to port 25 without any issues. The following commands yield the below results:
On my local machine:
$ telnet 10.8.0.1 25
Trying 10.8.0.1...
Connected to 10.8.0.1.
Escape character is '^]'.
220 xxxxx.com ESMTP Postfix (Debian/GNU)
quit
221 2.0.0 Bye
Connection closed by foreign host.
$ telnet 10.8.0.1 7462
Trying 10.8.0.1...
Connected to 10.8.0.1.
Escape character is '^]'.
Connection closed by foreign host.
$
On the remote server:
$ netstat -plnt | grep -P "7462|25"
tcp 0 0 0.0.0.0:25 0.0.0.0:* LISTEN -
tcp 0 0 10.8.0.1:7462 0.0.0.0:* LISTEN 6463/java
tcp6 0 0 :::25 :::* LISTEN -
$ telnet 10.8.0.1 7462
Trying 10.8.0.1...
Connected to 10.8.0.1.
Escape character is '^]'.
exit
OK Goodbye
Connection closed by foreign host.
An iptables -L command on both the local and remote machine show that I have no firewall rules established and I have not specifically configured any routes.
I can't figure out why I can connect to the remote server's port 25 from my local machine, but not port 7462. From the remote server though I can connect to port 7462 using the 10.8.0.1 IP address. I'd be happy to provide any additional information and thanks in advance for you help
-Nathan
I figured out my problem. The application was limiting the IP addresses that could connect on the 7462 port and it was a configuration item that needed to be set (default was just localhost). I feel a little silly for such a simple answer, but thanks for the suggestions!
I figured out my problem. The application was limiting the IP addresses that could connect on the 7462 port and it was an (obscure) configuration item that needed to be set (default was just localhost). I feel a little silly for such a simple answer, but thanks for the suggestions!