I have a spark application running on a remote server and I need to get its heap dump for performance purposes. I was able to run the jstatd service on the remote machine and connect to it using visualvm. However jstatd does not enable heap dump of remote machines (I am using visual vm 1.3.8).
To resolve this I started my application with the following extra options:
--conf "spark.driver.extraJavaOptions=-Dcom.sun.management.jmxremote \
-Dcom.sun.management.jmxremote.port=54321 \
-Dcom.sun.management.jmxremote.rmi.port=54320 \
-Dcom.sun.management.jmxremote.authenticate=false \
-Dcom.sun.management.jmxremote.ssl=false \
-Djava.rmi.server.hostname=$HOSTNAME"
After running this I used netstat to gather all open ports by the process and got the following output:
sudo netstat -lp | grep 37407
tcp 0 0 *:54321 *:* LISTEN 37407/java
tcp 0 0 *:54320 *:* LISTEN 37407/java
To check if the remote port was accessible through my local machine I used the netcat utility and the connection with remote host on both 54321 and 54320 was successful.
However when I try to connect to the host using visualvm or jconsole it fails to connect. Visual vm reports the following error:
cannot connect to hostname:54321 using service:jmx:rmi:///jndi/rmi://hostname:54321/jmxrmi
What am I doing wrong here?
in order to enable jconsole connection : try adding this flag
-Dcom.sun.management.jmxremote.local.only=false
and in order to a heap dump, you don't need to connect via jconsole, just use jmap:
$>jmap -dump:format=b,live,file=<filename> <process-id>
and finally, if spark has a daemon controlling it, make sure it doesn't kill the process during the heap dump creation.
The problem is that $HOSTNAME is the hostname of the server you are running spark submit from, you need to set to the hostname of the machine the spark driver runs on:
-Djava.rmi.server.hostname=<hostname of spark driver>
BTW, This is the reason it only worked for you when your spark application and the spark submit was on the same server.
Related
I have to tunnel the VNC server(tightvnc-server) running on my local machine to a remote server, so at remote server can access my local machine without port forwarding on router(at local machine network).
Right now I am using the following command
ssh -R 5950:localhost:5900 user#remote.ddns.net
Where 5900 the vnc server port on my local machine. And I have to access the machine from remote server using the command localhost:5950. And when I try to connect using vnc viewer I am getting the error connection closed unexpectedly. Normally if no connection exist I was getting connection refused error. But here something is missing in tunneling. Can any please tell me what could be the reason.
You need to activate the remote desktop, running:
sudo /System/Library/CoreServices/RemoteManagement/ARDAgent.app/Contents/Resources/kickstart \ -activate -configure -access -on \ -restart -agent -privs -all
Then you can connected using VNC Viewer
I am trying to transfer my console application written in ncurses under the terminal connected to the com port to the virtual terminal using telnetd.
When you try to start telnetd, you get: telnetd: getpeername: Socket operation on non-socket. When connected via putty, the connection crashes. netstat shows:
netstat
telnetd expects to be started via inetd. See http://www.qnx.com/developers/docs/7.0.0/index.html#com.qnx.doc.neutrino.user_guide/topic/tcpip_Daemons.html for information on how to set up inetd on QNX.
I created the Ubuntu virtual machine on that I created the Go environment after that I successfully imported and installed the skx/dns-api-go
I followed this doc: https://github.com/skx/dns-api-go for to set up the environment also I installed the required dependencies like
Dancer,Plack::Middle ware::Throttle::Lite,Net::DNS::Resolver,twiggy,
libdancer-perl ,libnet-cidr-lite-perl ,
libplack-middleware-reverseproxy-perl.
after that, at last, I run the following command:
dns-api-go -redis-server localhost:6379
then am getting the following result :
Launching the server on http://127.0.0.1:9999 redis: 2018/09/25
11:18:56 ring.go:263: ring shard state changed: Redis is down.
when I changed the port number in web preview in cloud shell it is showing as your server is not listing to 6379 port number, but in the instance level when I run the netstat -an command it is showing as the server is listing for 6379 port
please help me in resolving these issue.
6379 is redis server port which is there for rate limiting feature. If you go through the main.go file, you'll see go server port is 9999. Just run "dns-api-go" command and hit http://localhost:9999/
Though browsing several websites and here on stack overflow, there seems to be a way to view the messages in an Activemq queue using Jolokia and Hawt.io, but I have been unsuccessful to this point.
We are running our Activemq (version 5.12.0) as in embedded service in our Spring Webapp and exposed the Jolokia web services as explained in this webpage:
https://jolokia.org/reference/html/agents.html#agent-war-programmatic
When looking that the Jolokia web services via Hawt.io, I can not figure out how to actually view the messages in the queue.
Here is a screenshot showing the queue size:
So, how can I view the messages in an Activemq queue using Jolokia and Hawt.io?
The solution we ended up going with didn't actually use Jolokia or Hawt.io.
We ended up using Jconsole.
When looking at ActiveMQ queues, if you used a java serialized object in the queue, the data won't be very readably, but if you serialize your object to json, it is quite easy to see what is in the queue.
It is terribly important to read these directions all the way though, carefully.
These instructions discuss SSH Tunneling and it is quite easy to mess something up and there are not very good log messages when things go wrong.
Remote Debugging
Due to security reasons, we have closed all the open debug ports on our remote virtual machines.
To get remote debugging to work, we will need to use SSH Tunneling to access the remote virtual machine debugging ports.
Remote Application Setup
The application that you want to remotely debug must have the JPDA Transport connector enabled.
After Java 1.4, to enable the JPDA Transport, add the following vm parameter when starting your java virtual machine:
-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=<remote_port_number>
The above attributes are hard to describe, but what is presented above works well. More information about the above attributes can be found on the Connection and Invocation Details page.
Local IDE Setup
In Intellij to connect to a remote java virtual machine, open the "Run/Debug Configurations" window.
Then select a new "Remote" configuration.
Enter the following values:
Debugger mode
Attach to remote JVM
Host
localhost
Port
<local_port_number>*
Use module classpath
<local_package>**
The <port_number> should be the local port number of the ssh tunneling session that you will be starting. It is recommended that the <remote_port_number> and the <local_port_number> are the same value.
** This value should be whatever your local project is named.
SSH Tunneling
To actually connect to the remote debugging port, we'll need to use SSH Tunneling.
Run the following command via a terminal command line:
$ ssh -L <local_port_number>:localhost:<remote_port_number> -f <username>#<remote_server_name> -N
Example:
$ ssh -L 10001:localhost:10001 -f <your_username>#<your.server.com> -N
This command does the following:
Starts an ssh session with the <remote_server_name>.
Connects your <local_port_number> to the <remote_port_number> of the localhost of the remote machine. In this case, we're saying connect to localhost:10001 of the <your.server.com> machine.
Start remote debugging in the Intellij IDE and you should then be connected to the remote java virtual machine.
Resources
Intellij IDEA remotely debug java console program
Remote debug of a Java App using SSH tunneling (without opening server ports)
Remote JMX
We use JMX to look at the Spring Integration Kaha DB Queues.
Remote Application Setup
Add the following vm parameters:
-Dcom.sun.management.jmxremote.port=64250
-Dcom.sun.management.jmxremote.rmi.port=64250
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
-Djava.rmi.server.hostname=127.0.0.1
The jmxremote.port and jmxremote.rmi.port can be any number and they can be different values, it just helps if they are the same value when doing the ssh tunneling below.
SSH Tunneling
$ ssh -L 64250:localhost:64250 -f <your_username>#<your.server.com> -N
JConsole Setup
This is done in a new terminal window.
$ jconsole -J-DsocksProxyHost=localhost -J-DsocksProxyPort=64250 service:jmx:rmi:///jndi/rmi://127.0.0.1:64250/jmxrmi
Resources
Why Java opens 3 ports when JMX is configured?
Clean Up
To close the ssh processes above:
$ lsof -i tcp | grep ^ssh
Then perform a kill on the process id.
Using jps and jstack to Help Debug
List all java processes running on a machine:
$ sudo jps
List the threads of an application running:
$ sudo -u <process_owner> jstack <process_id>
Example:
$ sudo -u tomcat jstack <pid>
I want to debug a process running on a remote box from my host box (I built the code on the host machine).
Both have linux type operating systems.
I seems I can only communicate to the remote box from the host box via ssh (I tested using telnet).
I have followed the following steps to set this up:
On the Remote box:
Stop the firewall service:
service firewall_service stop
Attach the process to gdbserver
--attach :remote_port process_id
On the Host box:
Set up port forwarding via ssh
sudo ssh remote_username#remote_ip -L host_port:localhost:remote_port
-f sleep 60m
Set up gdb to attach to a remote process:
gdb file.debug
(gdb) target remote remote_ip:remote_port
When I try to start the debugging on the host by running 'target remote remote_ip:remote_port' on the host box I get a 'Connection timedout' error.
Can you guys see anything I am doing wrong, anything to check or an alternative way to debug remotely over ssh I would be grateful.
Thanks
This command:
sudo ssh remote_username#remote_ip -L host_port:localhost:remote_port ...
forwards local host_port to remote_port on remote_ip's localhost. This is useful only if you could not just connect to remote_ip:remote_port directly (for example, if that port is blocked by firewall).
This command:
(gdb) target remote remote_ip:remote_port
asks GDB to connect to remote_port on remote_ip. But you said that you can only reach remote_ip via ssh, so it's not surprising that GDB times out.
What you want:
ssh remote_username#remote_ip -L host_port:localhost:remote_port ...
(gdb) target remote :host_port
In other words, you connect to local host_port, and ssh forwards that local connection to remote_ip:remote_port, where gdbserver is listening for it.