Connecting with CLI in default way towards new Jenkins instance hangs.
java -jar $JENKINS_HOME/war/WEB-INF/jenkins-cli.jar -s http://git-dapp902.XXXXXXXX:17801 -auth admin:aaaaaaa who-am-i
(aaaaaa = initial password for admin account)
Following actions were performed
- startup new Jenkins instance
- log on as admin with initial password
- enable global security JLNP on fixed port (without this, I would get a Connection Refused)
- restart the jenkins server
- launch the above java command (on the same Linux VM)
- client does not return. No logging on Jenkins server
- When client is canceled below logging is shown
java -jar $JENKINS_HOME/war/WEB-INF/jenkins-cli.jar -s http://git-dapp902.XXXXXXXX:17801 -auth admin:aaaaaaa who-am-i
WARNING: Connection #4 failed
java.io.EOFException
at java.base/java.io.DataInputStream.readFully (DataInputStream.java:202)
at java.base/java.io.DataInputStream.readUTF(DataInputStream.java:614)
at java.base/java.io.DataInputStream.readUTF(DataInputStream.java:569)
at hudson.TcpSlaveAgentListener$ConnectionHandler.run(TcpSlaveAgentListener.java:254)
In the meantime I found the solution.
The Jenkins URL does not need to point to the port in JLNP setting, but to the "plain" port.
Luc
Related
Background
I have a machine in production running an elixir application (no access to iex, only to erl) and I am tasked with running an analysis on why we are consuming so much CPU. The idea here would be to launch observer, check the processes tab and see the processes with the most reductions.
How am I connecting?
To connect I am following a tutorial from a blog:
https://sgeos.github.io/elixir/erlang/observer/2016/09/16/elixir_erlang_running_otp_observer_remotely.html 1
Their instructions are as follows:
launch the app in the production machine with a cookie and a name
from local run: ssh user#public_ip "epmd -names" to get the name of the app and the port used
from local create a ssh tunnel to the remote machine: ssh -L 4369:user#public_ip:4369 -L 42877:user#public_ip:42877 user#public_ip (4369 is the epmd port by default, 42877 is the port of the app)
from local connect to the remote machine using the node's name: erl -name "user#app_name" -setcookie "mah_cookie" -hidden -run observer
Problem
And now in theory I should be able to use observer on the machine. Instead however I am greeted with the following error:
Protocol ‘inet_tcp’: register/listen error: epmd_close
So, after scouring the dark side of internet, I decided to use sudo journalctl -f to check all the logs of the machine and I found this:
channel 3: open failed: administratively prohibited: open failed
my_app_name sshd[8917]: error: connect_to flame#99.999.99.999: unknown host (Name or service not known)
/scripts/watchdog.sh")
my_app_name CRON[9985]: pam_unix(cron:session): session closed for user flame
Where:
erlang -name: my_app_name
machine user: flame
machine public ip: 99.999.99.999 (obviously not real)
so it tells me, unknown host ?? I am confused since 99.999.99.999 is the public IP of the machine itself!
Questions
What am I doing wrong?
I read that in older versions of erlang I can’t monitor a machine with observer if they are in different networks (which is the case, because I want to monitor this machine from my localhost) but I didn’t find any information regarding this in modern days.
If this is in fact impossible, what alternatives do I have?
Solution
After 3 days of non-stop searching, I finally found something that works.
To summarize I am putting it here everything I did.
All steps in local machine:
get the ports from the remote server:
> ssh remote-user#remote-ip "epmd -names"
epmd: up and running on port 4369 with data:
name super_duper_app at port 43175
create a ssh tunel with the ports:
ssh remote-user#remote-ip -L4369:localhost:4369 -L43175:localhost:43175
On another terminal in your local machine, run a iex terminal with the cookie the app in your remote server is using. Then connect to it and start observer:
iex --name observer#127.0.0.1 --cookie super_duper_cookie
Node.connect :"super_duper_app#127.0.0.1"
> true
:observer.start
With observer started, select the machine from the Nodes menu.
Possible setbacks
If you have tried this and it didn't work there are a few things you can check for:
Check if the EPMD port on your local machine is free, if not, kill the process using it and free it.
Check your ssh tunneling keys and configurations for permissions. As #Roberto Aloi pointed out this link can be useful: https://unix.stackexchange.com/questions/14160/ssh-tunneling-error-channel-1-open-failed-administratively-prohibited-open
I want to connect from my home pc with VPN (ubuntu-18.04) to work pc (ubuntu 18.04) using x2go.
Server side installation:
sudo add-apt-repository ppa:x2go/stable
sudo apt-get update
sudo apt-get install x2goserver x2goserver-xsession
Client side installation:
sudo apt-get install x2goclient
I am able to ssh to my remote desktop, but x2go doesn't work. Problem seems to be in the server side, since I tried to x2go connect from different computer (CentOs) and error was the same.
Client session preferences window:
I tried almost everything in session preferences, even with RSA key (I have it) or single application session type and many else, always the same error.
Client debug log: (x2goclient --debug)
x2go-INFO-1> "Starting X2Go Client 4.1.2.1..." x2go-WARNING-1> English
language requested, not loading translator. x2go-WARNING-1> English
language requested, not loading translator. Object::connect: No such
slot ONMainWindow::slotCheckAgentProcess() x2go-INFO-3> "Started X2Go
Client." x2go-DEBUG-../src/onmainwindow.cpp:563> "$HOME=/home/XXX"
x2go-DEBUG-../src/onmainwindow.cpp:2242> Reading 1 sessions from
config file. x2go-DEBUG-../src/sessionbutton.cpp:342> Creating QPixmap
with session icon: ":/img/icons/128x128/x2gosession.png".
x2go-DEBUG-../src/onmainwindow.cpp:12843> libssh not initialized yet.
Initializing. x2go-DEBUG-../src/onmainwindow.cpp:2728> Creating
QPixmap with session icon: '":/img/icons/128x128/x2gosession.png"'.
x2go-DEBUG-../src/onmainwindow.cpp:2795> Starting session via Smart
Card, SSH Agent or Kerberos token. x2go-INFO-8> "Starting connection
to server: 192.168.29.15:22" x2go-DEBUG-../src/onmainwindow.cpp:2829>
Starting new ssh connection to server:"192.168.29.15":"22" krbLogin:
false x2go-DEBUG-../src/sshmasterconnection.cpp:175>
SshMasterConnection, host "192.168.29.15"; port 22; user "XXX";
useproxy false; proxyserver "192.168.29.15"; proxyport 22
x2go-DEBUG-../src/sshmasterconnection.cpp:248> Starting SSH connection
without Kerberos authentication.
x2go-DEBUG-../src/sshmasterconnection.cpp:252> SshMasterConnection,
instance SshMasterConnection(0x56102247dbe0) created.
x2go-DEBUG-../src/sshmasterconnection.cpp:520> SshMasterConnection,
instance SshMasterConnection(0x56102247dbe0) entering thread.
x2go-DEBUG-../src/sshmasterconnection.cpp:840> Session port before
config file parse: 22 x2go-DEBUG-../src/sshmasterconnection.cpp:850>
Session port after config file parse: 22
x2go-DEBUG-../src/sshmasterconnection.cpp:915> Session port before
config file parse (part 2): 22
x2go-DEBUG-../src/sshmasterconnection.cpp:925> Session port after
config file parse (part 2): 22
x2go-DEBUG-../src/sshmasterconnection.cpp:950> cserverAuth
x2go-DEBUG-../src/sshmasterconnection.cpp:991> state: 1
x2go-DEBUG-../src/sshmasterconnection.cpp:1364> userAuthAuto failed:""
(code 1)
x2go-DEBUG-../src/sshmasterconnection.cpp:1274> Trying password
mechanism if available.
x2go-DEBUG-../src/sshmasterconnection.cpp:1278> Password mechanism
available. Continuing.
x2go-DEBUG-../src/sshmasterconnection.cpp:726> User authentication OK.
x2go-DEBUG-../src/sshmasterconnection.cpp:1789> LOGIN CHECK:"LOGIN OK
" x2go-DEBUG-../src/sshmasterconnection.cpp:1792> don't have
interaction x2go-DEBUG-../src/sshmasterconnection.cpp:1825> LOOP
FINISHED x2go-DEBUG-../src/sshmasterconnection.cpp:1829> No
interaction needed, continue session
x2go-DEBUG-../src/sshmasterconnection.cpp:735> Login Check - OK
x2go-DEBUG-../src/onmainwindow.cpp:2923> SSH connection established.
x2go-DEBUG-../src/onmainwindow.cpp:3336> Continue normal X2Go session
x2go-DEBUG-../src/sshprocess.cpp:199> Executing remote command via
SshProcess object 0: "x2golistsessions"
x2go-DEBUG-../src/sshprocess.cpp:213> this=SshProcess(0x5610224a1f30)
Running masterCon->addChannelConnection(this, '
"1d4dfc1a-7f4d-4f00-b39e-07273d3d973d" ', ' "bash -l -c 'echo
"X2GODATABEGIN:1d4dfc1a-7f4d-4f00-b39e-07273d3d973d"; export
PATH="/usr/local/bin:/usr/bin:/bin";export TERM="dumb";
x2golistsessions; echo "X2GODATAEND:1d4dfc1a-7f4d-4f00-b39e-07273d"
'); x2go-DEBUG-../src/sshmasterconnection.cpp:1891> Locking SSH
channel connection MUTEX.
x2go-DEBUG-../src/sshmasterconnection.cpp:1893> Passing new channel
connection object to channelConnections.
x2go-DEBUG-../src/sshmasterconnection.cpp:1895> Unlocking SSH channel
connection MUTEX. x2go-DEBUG-../src/sshmasterconnection.cpp:2083>
Creating new channel.
x2go-DEBUG-../src/sshmasterconnection.cpp:2100> New
channel:0x7fcf84012aa0
x2go-DEBUG-../src/sshmasterconnection.cpp:2185> Executing remote:
"bash -l -c 'echo
"X2GODATABEGIN:1d4dfc1a-7f4d-4f00-b39e-07273d3d973d"; export
PATH="/usr/local/bin:/usr/bin:/bin";export TERM="dumb";
x2golistsessions; echo
"X2GODATAEND:1d4dfc1a-7f4d-4f00-b39e-07273d3d973d";'"
x2go-DEBUG-../src/sshmasterconnection.cpp:2208> New exec channel
created.
x2go-DEBUG-../src/sshmasterconnection.cpp:2257> EOF on channel
0x7fcf84012aa0; SshProcess object: 0
x2go-DEBUG-../src/sshmasterconnection.cpp:2380> EOF sent.
x2go-DEBUG-../src/sshmasterconnection.cpp:2384> Channel closed.
x2go-DEBUG-../src/sshprocess.cpp:526> SSH finished: raw output
(stdout): "" x2go-DEBUG-../src/sshprocess.cpp:532> Have stderr only,
something must be wrong. x2go-DEBUG-../src/sshprocess.cpp:537> SSH
finished: false - "bash: bash: command not found " (0).
x2go-DEBUG-../src/onmainwindow.cpp:3804> "bash: bash: command not
found
If someone was interested, I can show server's side /var/log/syslog
Thank you for your help
Please try moving your .bashrc aside and retry.
It was my bad, silly me. I had a typo in /etc/environment file and I didn't notice it because I haven't restarted my PC after that typo. After fixing it, it worked.
OS: Mac OS Sierra
So I installed Jmeter and SSH sampler plugin. I want to access api host to send execute command on it via ssh. I am able to connect to this host via ssh using my ssh key, but can't do the same using Jmeter SSH Sampler.
Keep getting:
'Failed to connect to server: timeout: socket is not established'
.
Currently I'm out of ideas, how would I achieve this? Are there any peculiarities or additional settings I'm missing?
Adding screens with settings. I am filling in IP address of host, which I can access, just can't publish it.
The key is in OpenSSH format.
response_screen sampler_settings
Remote Commands: Linux/MacOSX
Linux, Unix and MacOSX operating systems can be remotely accessible (in majority of cases) through SSH (Secure Shell) protocol. To accomplish that
we can use the JMeter SSH Sampler plugin.
Installation:
Download ApacheJMeter_ssh-x.x.x.jar and jsch-x.x.x.jar from the SSH Sampler Releases page.
Drop Apache_JMeter_ssh-x.x.x.jar to /lib/ext folder of your JMeter installation.
Drop jsch-x.x.x.jar to /lib folder of your JMeter installation.
Restart JMeter.
You should see 2 new Samplers: SSH Command and SSH SFTP.
Your login can fail due to DDS key being rejected. As per OpenSSH 7.0 release notes:
Support for ssh-dss, ssh-dss-cert-* host and user keys is disabled
by default at run-time. These may be re-enabled using the
instructions at http://www.openssh.com/legacy.html
Alternatively you can specify path to id_rsa private key file
I don't think SSH Command sampler understands ~ shorthand, you should be providing full path to your SSH private key
You have to provide your SSH username as well
In any case make sure you can reach the port 22 of the machine where the SSH server is running from the machine where JMeter is running.
More information: How to Run External Commands and Programs Locally and Remotely from JMeter
I'm trying to follow beginner tutorial on Docker's website and I suffer with an error on login.
OS is Ubuntu 14.04, I'm not using VirtualBox and I'm not behind any proxy and want to push to the "regular" docker repository (not private one).
All threads I've found mention proxies and private repositories but that isn't my case, I'm just trying to do simple beginner tutorial.
Here is my attempt:
$ sudo docker login
[sudo] password for myuname:
Login with your Docker ID to push and pull images from Docker Hub. If you don't have a Docker ID, head over to https://hub.docker.com to create one.
Username: myDHuname
Password:
Error response from daemon: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
My docker info:
Containers: 5
Running: 0
Paused: 0
Stopped: 5
Images: 5
Server Version: 1.11.0
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 28
Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge null host
Kernel Version: 3.19.0-58-generic
Operating System: Ubuntu 14.04.4 LTS
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 7.686 GiB
Name: myuname-ThinkPad-T420
ID: 6RW3:X3FC:T62N:CWKI:JQW5:YIPY:RAHO:ZHF4:DFZ6:ZL7X:JPOD:V7EC
Docker Root Dir: /var/lib/docker
Debug mode (client): false
Debug mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Epilogue
Now docker login is passing. I have not touched anything since yesterday when it was broken...
I can't reproduce the behavior anymore.
I encounter this issue when my first use docker. I've shadowsocks proxy on, and configed as pac mode. When I try to run docker run hello-world, I get this timeout error. When I set the proxy mode to global, the error is aslo there.
But when I disable the proxy, docker runs well. It pull remote image successfully.
docker for windows
Note: Some users reported problems connecting to Docker Hub on Docker
for Windows stable version. This would manifest as an error when
trying to run docker commands that pull images from Docker Hub that
are not already downloaded, such as a first time run of docker run
hello-world. If you encounter this, reset the DNS server to use the
Google DNS fixed address: 8.8.8.8. For more information, see
Networking issues in Troubleshooting.
The error Client.Timeout exceeded while awaiting headers indicates:
GET request to the registry https://registry-1.docker.io/v2/ timedout
The library responsible (most likely libcurl) timed out before a response was heard
The connection never formed (proxy/firewall gobbled it up)
If you see the below result you can rule out timed out and network connectivity
$ curl https://registry-1.docker.io/v2/
{"errors":[{"code":"UNAUTHORIZED","message":"authentication required","detail":null}]}
If you get the above response next would be to check if your user environment has some proxy configuration.
env | grep "proxy"
Note: The docker runs as root. Maybe you have http_proxy in your env. Most likely I am wrong. Anywho see what happens with the curl GET request
change the proxy settings in the firefox. May be you are in access restricted mode. Just add the server address in the firefox settings -> preferences -> advanced -> network -> configuration (settings). Add the server ip in the no proxy for the issue can be resolved
Though browsing several websites and here on stack overflow, there seems to be a way to view the messages in an Activemq queue using Jolokia and Hawt.io, but I have been unsuccessful to this point.
We are running our Activemq (version 5.12.0) as in embedded service in our Spring Webapp and exposed the Jolokia web services as explained in this webpage:
https://jolokia.org/reference/html/agents.html#agent-war-programmatic
When looking that the Jolokia web services via Hawt.io, I can not figure out how to actually view the messages in the queue.
Here is a screenshot showing the queue size:
So, how can I view the messages in an Activemq queue using Jolokia and Hawt.io?
The solution we ended up going with didn't actually use Jolokia or Hawt.io.
We ended up using Jconsole.
When looking at ActiveMQ queues, if you used a java serialized object in the queue, the data won't be very readably, but if you serialize your object to json, it is quite easy to see what is in the queue.
It is terribly important to read these directions all the way though, carefully.
These instructions discuss SSH Tunneling and it is quite easy to mess something up and there are not very good log messages when things go wrong.
Remote Debugging
Due to security reasons, we have closed all the open debug ports on our remote virtual machines.
To get remote debugging to work, we will need to use SSH Tunneling to access the remote virtual machine debugging ports.
Remote Application Setup
The application that you want to remotely debug must have the JPDA Transport connector enabled.
After Java 1.4, to enable the JPDA Transport, add the following vm parameter when starting your java virtual machine:
-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=<remote_port_number>
The above attributes are hard to describe, but what is presented above works well. More information about the above attributes can be found on the Connection and Invocation Details page.
Local IDE Setup
In Intellij to connect to a remote java virtual machine, open the "Run/Debug Configurations" window.
Then select a new "Remote" configuration.
Enter the following values:
Debugger mode
Attach to remote JVM
Host
localhost
Port
<local_port_number>*
Use module classpath
<local_package>**
The <port_number> should be the local port number of the ssh tunneling session that you will be starting. It is recommended that the <remote_port_number> and the <local_port_number> are the same value.
** This value should be whatever your local project is named.
SSH Tunneling
To actually connect to the remote debugging port, we'll need to use SSH Tunneling.
Run the following command via a terminal command line:
$ ssh -L <local_port_number>:localhost:<remote_port_number> -f <username>#<remote_server_name> -N
Example:
$ ssh -L 10001:localhost:10001 -f <your_username>#<your.server.com> -N
This command does the following:
Starts an ssh session with the <remote_server_name>.
Connects your <local_port_number> to the <remote_port_number> of the localhost of the remote machine. In this case, we're saying connect to localhost:10001 of the <your.server.com> machine.
Start remote debugging in the Intellij IDE and you should then be connected to the remote java virtual machine.
Resources
Intellij IDEA remotely debug java console program
Remote debug of a Java App using SSH tunneling (without opening server ports)
Remote JMX
We use JMX to look at the Spring Integration Kaha DB Queues.
Remote Application Setup
Add the following vm parameters:
-Dcom.sun.management.jmxremote.port=64250
-Dcom.sun.management.jmxremote.rmi.port=64250
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
-Djava.rmi.server.hostname=127.0.0.1
The jmxremote.port and jmxremote.rmi.port can be any number and they can be different values, it just helps if they are the same value when doing the ssh tunneling below.
SSH Tunneling
$ ssh -L 64250:localhost:64250 -f <your_username>#<your.server.com> -N
JConsole Setup
This is done in a new terminal window.
$ jconsole -J-DsocksProxyHost=localhost -J-DsocksProxyPort=64250 service:jmx:rmi:///jndi/rmi://127.0.0.1:64250/jmxrmi
Resources
Why Java opens 3 ports when JMX is configured?
Clean Up
To close the ssh processes above:
$ lsof -i tcp | grep ^ssh
Then perform a kill on the process id.
Using jps and jstack to Help Debug
List all java processes running on a machine:
$ sudo jps
List the threads of an application running:
$ sudo -u <process_owner> jstack <process_id>
Example:
$ sudo -u tomcat jstack <pid>