I have a Bitnami Wordpress (single site) installed VM on Google Cloud instance that was working perfectly until I stopped/Started the instance.
After doing so, my website went down, and I can no longer SSH to it using the browser nor OSX Terminal. I get the following message:
"We are unable to connect to the VM on port 22"
I double checked the Firewall on Google cloud - which I did not change from default - and everything seems ok.
It is not Pingable and I even tried to trace route to the server but it reaches to 216.239.42.191 then stops before reaching my VM.
So I tried to connect using the serial Console and it is flooded with:
Sep 30 07:44:17 localhost kernel: [43306.942210] IPv4: martian source 10.128.0.2 from 74.125.73.34, on dev eth0
Sep 30 07:44:17 localhost kernel: [43306.949531] ll header: 00000000: 42 01 0a 80 00 02 42 01 0a 80 00 01 08 00
B.....B.......
[43317.271565] IPv4: martian source 10.128.0.2 from 10.128.0.1, on dev eth0
[43317.278571] ll header: 00000000: 42 01 0a 80 00 02 42 01 0a 80 00 01 08 06 B.....B.......
Sep 30 07:44:28 localhost kernel: [43317.271565] IPv4: martian source 10.128.0.2 from 10.128.0.1, on dev eth0
Sep 30 07:44:28 localhost kernel: [43317.278571] ll header: 00000000: 42 01 0a 80 00 02 42 01 0a 80 00 01 08 06
B.....B.......
[43377.265708] IPv4: martian source 10.128.0.2 from 10.128.0.1, on dev eth0
[43377.272733] ll header: 00000000: 42 01 0a 80 00 02 42 01 0a 80 00 01 08 06 B.....B.......
Sep 30 07:45:28 localhost kernel: [43377.265708] IPv4: martian source 10.128.0.2 from 10.128.0.1, on dev eth0
Sep 30 07:45:28 localhost kernel: [43377.272733] ll header: 00000000: 42 01 0a 80 00 02 42 01 0a 80 00 01 08 06
Any Idea?
1 - I would recommend that you read the GCP documentation on how to handle the "Unable to connect on port 22" error message.
More details on how to troubleshoot SSH issues and workarounds can be found in the following link.
2 - Since you’ve said that this occurred right after you’ve stopped/started the instance, I highly suspect that you’ve assigned an ephemeral external IP address to the VM instance.
If this is the case, I would suggest that you look at changing the VM instance IP address from an ephemeral IP address to a static IP address. More details on how to reserve a static External IP address to the VM instance can be found in this link.
3 - By looking at the serial console logs, I can see that the issue here is related to the packets having a Martian origin. So, every packet received on the VM, whatever was the source or destination, was just discarded by the Kernel for these packets being flagged as having a Martian source.
You may read more about it in this article. There are a few solutions offered. Alternatively, there are other resolution steps in this article.
Related
I'm using Fiddler Proxy to decrypt HTTPS traffic for a specific application, the problem I'm facing is the application seem to be using the internal browser to render part of the information and it seems when rendering to browser Fiddler is unable to tunnel even tho it's hitting the same hostname.. I've capture a successful and an unsuccessful connection but I'm not expert on this so hopping someone can give a hand and tell me if it's something that can be resolved somehow. To give complete info, I'm using jailbroken iPhone XR with SSLKillSwitch to bypass pinning certificate errors, it works for regular options in the App but when I get to the section where it uses the internal webkit browser then the tunnel closes the connections.
Here is a successful tunnel stablished:
CONNECT api.xxxxxxxxx.com:443 HTTP/1.1
Host: api.xxxxxxxx.com
User-Agent: Driver/1003.95.3.17728954 CFNetwork/1240.0.4 Darwin/20.6.0
Connection: keep-alive
Connection: keep-alive
A SSLv3-compatible ClientHello handshake was found. Fiddler extracted the parameters below.
Version: 3.3 (TLS/1.2)
Random: 53 36 A3 4D 40 7F 06 DC 59 EF 0D F2 67 BF 69 13 90 B3 40 A7 30 A1 14 54 E8 D0 ED 0D 99 78 66 05
"Time": 4/11/2011 1:11:47 PM
SessionID: 67 37 00 00 89 55 CA 97 B4 ED 1E 51 D3 52 84 D9 C0 95 92 E8 3E AA 22 59 39 ED EE 34 40 D5 26 96
Extensions:
grease (0xaaaa) empty
server_name api.xxxxxxxx.com
extended_master_secret empty
renegotiation_info 00
supported_groups grease [0xfafa], x25519 [0x1d], secp256r1 [0x17], secp384r1 [0x18], secp521r1 [0x19]
ec_point_formats uncompressed [0x0]
ALPN http/1.1
status_request OCSP - Implicit Responder
signature_algs ecdsa_secp256r1_sha256, rsa_pss_rsae_sha256, rsa_pkcs1_sha256, ecdsa_secp384r1_sha384, ecdsa_sha1, rsa_pss_rsae_sha384, rsa_pss_rsae_sha384, rsa_pkcs1_sha384, rsa_pss_rsae_sha512, rsa_pkcs1_sha512, rsa_pkcs1_sha1
SignedCertTimestamp (RFC6962) empty
key_share 00 29 FA FA 00 01 00 00 1D 00 20 63 5F C7 E5 45 CB 0C 1B 17 34 69 DF B4 F5 98 0C 91 23 A5 D8 C0 17 C9 8D CC 70 B8 23 C7 79 67 1A
psk_key_exchange_modes 01 01
supported_versions grease [0x2a2a], Tls1.3, Tls1.2
grease (0xcaca) 00
padding 214 null bytes
Ciphers:
[BABA] Unrecognized cipher - See https://www.iana.org/assignments/tls-parameters/
[1301] TLS_AES_128_GCM_SHA256
[1302] TLS_AES_256_GCM_SHA384
[1303] TLS_CHACHA20_POLY1305_SHA256
[C02C] TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
[C02B] TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
[CCA9] TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256
[C030] TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
[C02F] TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
[CCA8] TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256
[C024] TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384
[C023] TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256
[C00A] TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA
[C009] TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA
[C028] TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384
[C027] TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
[C014] TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
[C013] TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
Compression:
[00] NO_COMPRESSION
HTTP/1.1 200 Connection Established
FiddlerGateway: Direct
StartTime: 08:53:48.980
Connection: close
Encrypted HTTPS traffic flows through this CONNECT tunnel. HTTPS Decryption is enabled in Fiddler, so decrypted sessions running in this tunnel will be shown in the Web Sessions list.
Secure Protocol: Tls12
Cipher: Aes128 128bits
Hash Algorithm: Sha256 ?bits
Key Exchange: ECDHE_RSA (0xae06) 255bits
== Server Certificate ==========
[Subject]
CN=*.xxxxxx.com, O="Xxxxxx, Inc.", L=San Francisco, S=California, C=US
[Issuer]
CN=DigiCert TLS RSA SHA256 2020 CA1, O=DigiCert Inc, C=US
[Serial Number]
0E5C9FB26125F869BF32DEFE4B26822E
[Not Before]
6/13/2022 8:00:00 PM
[Not After]
7/15/2023 7:59:59 PM
[Thumbprint]
4F91631510EC84B84A195014E335B1C6748318AF
[SubjectAltNames]
*.xxxxxx.com, xxxxx.com, *.xxxxx.net, xxxxx.net, *.xxxxx.me, xxxxx.me, *.xxxxx.ca, xxxxx.ca, *.xxxxxxxxx.com, xxxxxxxx.com
And tunnel unsuccessful:
CONNECT api.xxxxxx.com:443 HTTP/1.1
Host: api.xxxxxxx.com
User-Agent: com.apple.WebKit.Networking/8611.4.1.0.3 CFNetwork/1240.0.4 Darwin/20.6.0
Connection: keep-alive
Connection: keep-alive
A SSLv3-compatible ClientHello handshake was found. Fiddler extracted the parameters below.
Version: 3.3 (TLS/1.2)
Random: E2 EF 8D 53 08 4F D9 3E 23 DB BA 45 40 A6 A2 ED 6E 23 3B 84 C6 00 98 75 9F 03 5C 95 6C 7E 6B 1E
"Time": 6/3/2014 11:55:14 AM
SessionID: CA 79 37 63 83 57 8B E1 86 24 8F F0 18 FA A9 27 83 52 1E 5B BD 39 27 86 94 CB 54 68 7D 7B FD 3E
Extensions:
grease (0x1a1a) empty
server_name api.xxxxxx.com
extended_master_secret empty
renegotiation_info 00
supported_groups grease [0xcaca], x25519 [0x1d], secp256r1 [0x17], secp384r1 [0x18], secp521r1 [0x19]
ec_point_formats uncompressed [0x0]
ALPN h2, http/1.1
status_request OCSP - Implicit Responder
signature_algs ecdsa_secp256r1_sha256, rsa_pss_rsae_sha256, rsa_pkcs1_sha256, ecdsa_secp384r1_sha384, ecdsa_sha1, rsa_pss_rsae_sha384, rsa_pss_rsae_sha384, rsa_pkcs1_sha384, rsa_pss_rsae_sha512, rsa_pkcs1_sha512, rsa_pkcs1_sha1
SignedCertTimestamp (RFC6962) empty
key_share 00 29 CA CA 00 01 00 00 1D 00 20 2C 81 5B 83 4E A9 2F E0 17 99 47 E1 51 C3 88 5E 6C 65 3C F6 FF FD DE BD B6 4F 3F 38 73 DB 1F 15
psk_key_exchange_modes 01 01
supported_versions grease [0xdada], Tls1.3, Tls1.2
grease (0x8a8a) 00
padding 211 null bytes
Ciphers:
[EAEA] Unrecognized cipher - See https://www.iana.org/assignments/tls-parameters/
[1301] TLS_AES_128_GCM_SHA256
[1302] TLS_AES_256_GCM_SHA384
[1303] TLS_CHACHA20_POLY1305_SHA256
[C02C] TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
[C02B] TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
[CCA9] TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256
[C030] TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
[C02F] TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
[CCA8] TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256
[C024] TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384
[C023] TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256
[C00A] TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA
[C009] TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA
[C028] TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384
[C027] TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
[C014] TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
[C013] TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
Compression:
[00] NO_COMPRESSION
HTTP/1.1 200 Connection Established
FiddlerGateway: Direct
StartTime: 08:53:46.605
Connection: close
Edited to add the error I get in Fiddler Log:
08:53:46:7774 !SecureClientPipeDirect failed: System.IO.IOException Authentication failed because the remote party has closed the transport stream. for pipe (CN=*.xxxxxxx.com, O=DO_NOT_TRUST, OU=Created by http://www.fiddler2.com)
Notice I've edited the actual hostname and developer info on purpose for privacy but left everything else untouched.. you can see different user-agent showing when it's using the browser and failing and when it's not.
Hope someone can give some clues on how to address. By using a different proxy (Mockttp) this works fine so I'm hoping Fiddler can also do it since it's more user friendly for my purpose.
Edited to clarify that I'm using SSL Kill Switch 2 (0.14-3+debug)
Edit2: It turns out my version of SSL Kill Switch 2 was older than latest and some fixes were now in place. I updated and now there is no certificate errors anymore, however, after the request worked for a couple of times I started getting 403 which initially I thought it was my IP being denied but then testing with Postman I get 200 all the time, I'm only denied when using Fiddler Proxy so still trying to figure out why the difference. I'm making sure Postman is sending exactly all the same headers and message as the app from the phone. Comparing a working vs denied from the phone doesn't seem to show any differences either.
Thanks!
Thanks to #Robert who pointed out it could be an issue with SSL Killswitch rather than Fiddler Proxy and after upgrading to a more recent version I no longer get the tunnel issue.. now I'm having a different problem which seems to be related to TLS fingerprint which I think I was able to confirm by using different proxy. So in the interest of getting this question as closed I can say #Robert help and comments were spot on!
This Redis Cluster have 240 nodes (120 masters and 120 slaves), and works well for a long time. But now it get a Master Slave switch almost several hours.
I get some log from Redis Server.
5c541d3a765e087af7775ba308f51ffb2aa54151
10.12.28.165:6502
13306:M 08 Mar 18:55:02.597 * Background append only file rewriting started by pid 15396
13306:M 08 Mar 18:55:41.636 # Cluster state changed: fail
13306:M 08 Mar 18:55:45.321 # Connection with slave client id #112948 lost.
13306:M 08 Mar 18:55:46.243 # Configuration change detected. Reconfiguring myself as a replica of afb6e012db58bd26a7c96182b04f0a2ba6a45768
13306:S 08 Mar 18:55:47.134 * AOF rewrite child asks to stop sending diffs.
15396:C 08 Mar 18:55:47.134 * Parent agreed to stop sending diffs. Finalizing AOF...
15396:C 08 Mar 18:55:47.134 * Concatenating 0.02 MB of AOF diff received from parent.
15396:C 08 Mar 18:55:47.135 * SYNC append only file rewrite performed
15396:C 08 Mar 18:55:47.186 * AOF rewrite: 4067 MB of memory used by copy-on-write
13306:S 08 Mar 18:55:47.209 # Cluster state changed: ok
5ac747878f881349aa6a62b179176ddf603e034c
10.12.30.107:6500
22825:M 08 Mar 18:55:30.534 * FAIL message received from da493af5bb3d15fc563961de09567a47787881be about 5c541d3a765e087af7775ba308f51ffb2aa54151
22825:M 08 Mar 18:55:31.440 # Failover auth granted to afb6e012db58bd26a7c96182b04f0a2ba6a45768 for epoch 323
22825:M 08 Mar 18:55:41.587 * Background append only file rewriting started by pid 23628
22825:M 08 Mar 18:56:24.200 # Cluster state changed: fail
22825:M 08 Mar 18:56:30.002 # Connection with slave client id #382416 lost.
22825:M 08 Mar 18:56:30.830 * FAIL message received from 0decbe940c6f4d4330fae5a9c129f1ad4932405d about 5ac747878f881349aa6a62b179176ddf603e034c
22825:M 08 Mar 18:56:30.840 # Failover auth denied to d46f95da06cfcd8ea5eaa15efabff5bd5e99df55: its master is up
22825:M 08 Mar 18:56:30.843 # Configuration change detected. Reconfiguring myself as a replica of d46f95da06cfcd8ea5eaa15efabff5bd5e99df55
22825:S 08 Mar 18:56:31.030 * Clear FAIL state for node 5ac747878f881349aa6a62b179176ddf603e034c: slave is reachable again.
22825:S 08 Mar 18:56:31.030 * Clear FAIL state for node 5c541d3a765e087af7775ba308f51ffb2aa54151: slave is reachable again.
22825:S 08 Mar 18:56:31.294 # Cluster state changed: ok
22825:S 08 Mar 18:56:31.595 * Connecting to MASTER 10.12.30.104:6404
22825:S 08 Mar 18:56:31.671 * MASTER SLAVE sync started
22825:S 08 Mar 18:56:31.671 * Non blocking connect for SYNC fired the event.
22825:S 08 Mar 18:56:31.672 * Master replied to PING, replication can continue...
22825:S 08 Mar 18:56:31.673 * Partial resynchronization not possible (no cached master)
22825:S 08 Mar 18:56:31.691 * AOF rewrite child asks to stop sending diffs.
It appends that Redis Master Slave Swtich happend after Aof rewtiting.
Here is the config of this cluster.
daemonize no
tcp-backlog 511
timeout 0
tcp-keepalive 60
loglevel notice
databases 16
dir "/var/cachecloud/data"
stop-writes-on-bgsave-error no
repl-timeout 60
repl-ping-slave-period 10
repl-disable-tcp-nodelay no
repl-backlog-size 10000000
repl-backlog-ttl 7200
slave-serve-stale-data yes
slave-read-only yes
slave-priority 100
lua-time-limit 5000
slowlog-log-slower-than 10000
slowlog-max-len 128
hash-max-ziplist-entries 512
hash-max-ziplist-value 64
list-max-ziplist-entries 512
list-max-ziplist-value 64
set-max-intset-entries 512
zset-max-ziplist-entries 128
zset-max-ziplist-value 64
activerehashing yes
client-output-buffer-limit normal 0 0 0
client-output-buffer-limit slave 512mb 128mb 60
client-output-buffer-limit pubsub 32mb 8mb 60
hz 10
port 6401
maxmemory 13000mb
maxmemory-policy volatile-lru
appendonly yes
appendfsync no
appendfilename "appendonly-6401.aof"
dbfilename "dump-6401.rdb"
aof-rewrite-incremental-fsync yes
no-appendfsync-on-rewrite yes
auto-aof-rewrite-min-size 62500kb
auto-aof-rewrite-percentage 86
rdbcompression yes
rdbchecksum yes
repl-diskless-sync no
repl-diskless-sync-delay 5
maxclients 10000
hll-sparse-max-bytes 3000
min-slaves-to-write 0
min-slaves-max-lag 10
aof-load-truncated yes
notify-keyspace-events ""
bind 10.12.26.226
protected-mode no
cluster-enabled yes
cluster-node-timeout 15000
cluster-slave-validity-factor 10
cluster-migration-barrier 1
cluster-config-file "nodes-6401.conf"
cluster-require-full-coverage no
rename-command FLUSHDB ""
rename-command FLUSHALL ""
rename-command KEYS ""
In my option, aof rewrite will not effect the Redis Main Thread. BUT this seems make this node not response other nodes' Ping.
Check THP(Transparent Huge pages) on Linux kernel parameter.
because AOF diff size 0.02MB, copy-on-write size 2067MB.
I have a bunch of Tomcat 8.0.15, Apache 2.2.29 and mod_jk 1.2.40 (win2003 serv)
In the http response status text I get "200 ACT". Is not a standart http 1.1 rfc "200 OK".
It doesn't affect on a normal work, but response is really weird...
Also tested on a clean-default configuration tomcat8-apache2.4(win7) - the same result.
In the mod_jk debug log we have:
[debug] ajp_connection_tcp_get_message::jk_ajp_common.c (1403): received from ajp13 pos=0 len=69 max=8192
[debug] ajp_connection_tcp_get_message::jk_ajp_common.c (1403): 0000 04 00 C8 00 03 41 43 54 00 00 02 00 0D 43 61 63 - .....ACT.....Cac
Normal response is like:
[debug] ajp_connection_tcp_get_message::jk_ajp_common.c (1403): received from ajp13 pos=0 len=100 max=8192
[debug] ajp_connection_tcp_get_message::jk_ajp_common.c (1403): 0000 04 00 C8 00 02 4F 4B 00 00 04 00 04 45 54 61 67 - .....OK.....ETag
Rainer Jung has fixed this in the 8.0.17 Tomcat release.
Explanation: http://svn.apache.org/viewvc?view=revision&revision=1645245
Patch:
--- java/org/apache/coyote/ajp/AjpProcessor.java (revision 1645245)
+++ java/org/apache/coyote/ajp/AjpProcessor.java (working copy)
## -1388,6 +1388,7 ##
response.setCommitted(true);
+ tmpMB.recycle();
responseMsgPos = -1;
responseMessage.reset();
responseMessage.appendByte(Constants.JK_AJP13_SEND_HEADERS)
Is working now, Already tested.
I configured WebSphere MQ v6.0.1.1 to be accessed by a Java client using JNDI and JMS via SSL. I tried the client and server on different machine, and on the same machine. I didn't get the same exception on the client side but it's related to a connection problem. On the server side I have nothing in the log.
Different machine client side error: Thread pool thread #0, Exception sending alert: java.net.SocketException: Software caused connection abort: socket write error
*** ServerHelloDone
*** Certificate chain
***
*** ClientKeyExchange, RSA PreMasterSecret, TLSv1
Thread pool thread #0, WRITE: TLSv1 Handshake, length = 141
SESSION KEYGEN:
PreMaster Secret:
0000: 03 01 DB 7F 1B 78 46 24 D1 B3 7F 8F E4 2B 2D 35 .....xF$.....+-5
0010: 1B EB FF C9 01 C9 EC 12 07 0F F9 88 A9 12 45 77 ..............Ew
0020: 22 AE 79 17 C2 9D 4C 97 04 3E BA 91 1F 14 68 44 ".y...L..>....hD
CONNECTION KEYGEN:
Client Nonce:
0000: 50 76 7B FB 0D 45 F0 8D EF 54 E0 AB 2C 3A D4 7D Pv...E...T..,:..
0010: 24 52 FB FB 4F F4 1D E4 CC 2C 4E BA 8B CA 3E 54 $R..O....,N...>T
Server Nonce:
0000: 00 00 00 00 8F 53 C4 4D 2F 2F 41 AA EB 0A 80 2D .....S.M//A....-
0010: D0 E4 51 2A CC BC EE 94 92 BD CD E0 9B C9 EB 3D ..Q*...........=
Master Secret:
0000: 9D 93 ED F3 8A 97 39 7F 71 5F 34 52 30 A6 8E 38 ......9.q_4R0..8
0010: BC 17 59 28 78 63 AA 66 63 D0 EE 1C C6 54 CA D1 ..Y(xc.fc....T..
0020: F2 F0 ED 7E D7 81 33 C6 E3 1B 7C 46 C0 FB C8 5C ......3....F...\
Client MAC write Secret:
0000: 57 56 3D 05 B1 27 BE 56 A8 FD 07 64 0A 96 62 F2 WV=..'.V...d..b.
0010: AE AF B5 98 ....
Server MAC write Secret:
0000: F5 C7 B2 D2 79 11 90 6C C8 FD 86 8B E5 AE 59 71 ....y..l......Yq
0010: B2 A7 AB D3 ....
Client write key:
0000: 54 FD FD 8B C2 B4 8B 3F 38 23 25 5A 8A 41 26 9B T......?8#%Z.A&.
Server write key:
0000: 6D 9C C0 97 ED 21 3F 0E 0A FB E2 2B EE C0 5F 01 m....!?....+.._.
... no IV used for this cipher
Thread pool thread #0, WRITE: TLSv1 Change Cipher Spec, length = 1
*** Finished
verify_data: { 182, 85, 56, 238, 250, 233, 155, 119, 224, 254, 23, 196 }
***
Thread pool thread #0, WRITE: TLSv1 Handshake, length = 36
Thread pool thread #0, READ: TLSv1 Change Cipher Spec, length = 1
Thread pool thread #0, READ: TLSv1 Handshake, length = 36
*** Finished
verify_data: { 215, 140, 30, 150, 82, 161, 85, 160, 127, 189, 226, 74 }
***
%% Cached client session: [Session-1, SSL_RSA_WITH_RC4_128_SHA]
Thread pool thread #0, setSoTimeout(120000) called
Thread pool thread #0, WRITE: TLSv1 Application Data, length = 150
Thread pool thread #0, READ: TLSv1 Application Data, length = 56
Thread pool thread #0, WRITE: TLSv1 Application Data, length = 48
Thread pool thread #0, called close()
Thread pool thread #0, called closeInternal(true)
Thread pool thread #0, SEND TLSv1 ALERT: warning, description = close_notify
Thread pool thread #0, WRITE: TLSv1 Alert, length = 22
Thread pool thread #0, Exception sending alert: java.net.SocketException: Software caused connection abort: socket write error
Same machine client side error: Thread pool thread #0, Exception sending alert: java.net.SocketException: Broken pipe
It seems that the client writes whereas the server has already closed the connection.
EDIT:
10/10/12 2:26:23 PM - Process(10995.12) User(mqm) Program(amqrmppa)
AMQ9631: The CipherSpec negotiated during the SSL handshake does not match the
required CipherSpec for channel 'SSL.CHANNEL'.
EXPLANATION:
There is a mismatch between the CipherSpecs on the local and remote ends of
channel 'SSL.CHANNEL'. The channel will not run until this mismatch is
resolved. The CipherSpec required in the local channel definition is
'RC4_SHA_US'. The name of the CipherSpec negotiated during the SSL handshake is
'RC4_SHA_US'. A code is displayed if the name of the negotiated CipherSpec
cannot be determined.
ACTION:
Change the channel definitions for 'SSL.CHANNEL' so the two ends have matching
CipherSpecs and restart the channel. If the certificate in use by one end of
the channel is a Global Server Certificate, then the negotiated CipherSpec may
not match that specified on either end of the channel. This is because the SSL
protocol allows a Global Server Certificate to automatically negotiate a higher
level of encryption. In these cases specify a CipherSpec which meets the
requirements of the Global Server Certificate.
----- amqccisa.c : 851 --------------------------------------------------------
10/10/12 2:26:23 PM - Process(10995.12) User(mqm) Program(amqrmppa)
AMQ9999: Channel program ended abnormally.
EXPLANATION:
Channel program 'SSL.CHANNEL' ended abnormally.
ACTION:
Look at previous error messages for channel program 'SSL.CHANNEL' in the error
files to determine the cause of the failure.
----- amqrmrsa.c : 468 --------------------------------------------------------
Edit 2:
1 : DIS CHANNEL(SSL.CHANNEL) SSLCIPH
AMQ8414: Display Channel details.
CHANNEL(SSL.CHANNEL) CHLTYPE(SVRCONN)
SSLCIPH(RC4_SHA_US)
The Cipher used client side using JMSAdmin:
DEFINE QCF(QCF_NAME) SYNCPOINTALLGETS(YES) HOSTNAME(HOST) PORT(1414) TRANSPORT(client) QMANAGER(MYQMGR) CHANNEL(SSL.CHANNEL) SSLCIPHERSUITE(SSL_RSA_WITH_RC4_128_SHA)
Base on SSL CipherSpecs and CipherSuites RC4_SHA_US seems to match SSL_RSA_WITH_RC4_128_SHA.
There is a possibility when running the client on the same host as the QMgr to connect using bindings mode (shared memory) rather than over the network stack. Since bindings mode connections do not use the network stack, SSL would make no difference.
Assuming that the client is connecting over the network in both cases, there's nothing about the location of the client on one server or another that would influence the SSL connection other than that the client configurations were different from one instance to the other. The client is still going through the network stack and presenting a connection request to the QMgr's listener. The client finds its keystore the same way and all the QMgr sees is a connection request presented to the listener. So if you are getting different results between the two client instances, look for configuration discrepancies.
My method for debugging SSL connections on WMQ is to progress through the following sequence making sure each step works before advancing to the next:
Get the channel running without SSL. This validates that the channel names are spelled correctly, that a network route exists between the endpoints, that the QMgr's listener is running and that the client points to the right port.
Get the channel running with the SVRCONN definition set to SSLCAUTH(OPTIONAL). This performs an anonymous SSL connection similar to what your browser does. The QMgr presents a certificate to the client but does not request one back. This validates that the QMgr can find its certificate and that the client can find its trust store and properly validates the QMgr's cert.
Set the SVRCONN channel to SSLCAUTH(REQUIRED). This now requires the client to find its keystore (in the last step it required only its trust store) and to be able to find its certificate. It also requires the QMgr to be able to validate the client's cert.
The difference between the last two steps helps to isolate the problem by testing the SSL credential exchange in only one direction at a time.
You mention that there's "nothing in the log" when this happens. Which log? There are two sets of error logs. One is the server-global log at {WMQ home}/errors and the other is the QMgr error log at {WMQ home}/QMgrs/{QMgr name}/errors. Error log entries are made to the QMgr's error log when the MQ can identify the QMgr associated with the error. However, when an SSL connection is requested, MQ does not know which QMgr the connection has requested until after the SSL connection has completed. Because of this, many SSL negotiation errors on the server side are reported in the global error log.
I'd suggest running through the steps outlined above to determine which SSL credential exchange is failing and then looking for the error log entries in both the QMgr and global error log files. If this does not resolve the problem, please update your question noting the step in the process that fails and any error log entries identified by which log they were found in.
Also, V6 of MQ went out of service as of last month. Moving to a supported version of WMQ client and server would allow you to open a PMR, would provide much better Java/JMS performance, and will allow you to use more secure ciphers such as SHA-2 hashes and the new elliptic curve crypto supported by GSKit 8. Since WMQ V6 is out of support, at most only one additional Fix Pack will be released, after which security vulnerabilities in that version will not be addressed. If you are using SSL, I assume security is somewhat important and that you would want to use a version that will receive fixes if a new vulnerability is discovered.
UPDATE
Responding to the question update regarding the Global Server Certificate, it is necessary to understand how WMQ implements SSL/TLS. When the connection is made, the TLS negotiation (if you specify an SSL , WMQ performs a TLS session using an SSL cipher) follows the spec and the spec allows for a negotiation of the ciphersuite. When a global server certificate is used, the cert can specify a minimum acceptable cipher strength and this influences the negotiation.
When the TLS session is completed successfully, the connection is then handed to WebSphere MQ. Only then does the QMgr check the channel parameters such as "what QMgr is the connection requested for?" or, more importantly, "does the connection's negotiated cipherspec match the channel definition?" Typically it fails based on a mismatch between client and server. With a global server certificate it can fail because of a mismatch between the negotiated cipherspec versus the minimum acceptable as specified by the certificate.
What the error message is saying is that it is possible for the ciphersuite specified by the client to exactly match the cipherspec specified in the channel and still fail to connect because the global server certificate specifies a minimum cipher strength that is greater than that used by the client and the QMgr. There is more on this at Cipherspec mismatches in the Infocenter but in this case the error message is almost as informative as the Infocenter.
I am wondering whether is it possible or not to establish a connection to a LDAP server via telnet (or some other program) and start making requests and receiving responses as I would normally do with HTTP. In fact, the question is more generic and is related to my misunderstanding of network connections and communications protocols. Let me tell you the idea I have in my mind about this topic:
All application protocols define communication protocols (that is, messages that the server is going to understand and act upon its delivery). If I know how the application protocol works, I can establish a connection to the server (daemon controlling that protocol server-side) and start communicating with the server. For example with HTTP I can establish a connection to an HTTP SERVER via telnet and start talking with him with this requests for example:
GET /users/pepito HTTP/1.1
Host: stackoverflow
Content-Type: text/html
I am expecting this procedure to happen with ANY APPLICATION PROTOCOL. Is this concept right??
I have glimpsed the LDAP Protocol Specification RFC but I did not understand the format of the messages. I mean, I was expecting to read something like HTTP Protocol Specification; but it was like too generic. Can you give me an example of how LDAP search could be made?
The LDAP RFC specifies that LDAP messages are ASN1 encoded. This means the messages are binary data in a special format, instead of text, following a special format. This makes it very hard to write ladap-queries by hand with telnet.
You can, somewhat, with a little help from some command-line friends :-)
Here's a hexdump of a simple LDAP query -- it does the equivalent of ldapsearch -x -b "" -s base objectclass=top:
30 0c 02 01 01 60 07 02 01 03 04 00 80 00
30 2c 02 01 02 63 27 04 00 0a 01 00 0a 01 00 02 01 00 02 01 1e 01 01 00 a3 12 04 0b 6f 62 6a 65 63 74 63 6c 61 73 73 04 03 74 6f 70 30 00
Save this to a file called ldap.hexdump, and then you can use nc:
xxd -r -p ldap.hexdump | nc ldapserver 389
If you want to see the output parsed, you can use unber:
xxd -r -p ldap.hexdump | nc ldapserver 389 | unber -m -
Where this might come in handy is if you can't use ldapsearch for some reason and want to use nc or openssl to test out whether an LDAP server is responding properly. It assumes that the server accepts anonymous binds to query the empty base DN (root DSE).