Who's Messing Up this TCP Connection? - embedded

I'm responsible for some embedded software that has to work with a customer's proprietary TCP interface (also embedded, but running under a well known and well regarded RTOS), but it's not getting through the three-way handshake, even though the HTTP interface, etc., all work fine, and I can communicate using the custom protocol with a program running on my PC.
Looking at the WireShark captures, his side initiates by sending a SYN, I send a SYN-ACK, and then he immediately sends a RST, so it looks like the problem is on his end. Is my analysis correct?
Here's a typical three packet example of the problem, with the MAC IDs anonymized (the real MAC IDs are valid). Sorry about pasting the raw hex, if anybody's got a better idea of how to put the WireShark capture up, I'm certainly amenable.
63 2009-06-29 13:07:49.685057 10.13.91.2 10.13.92.3 TCP 1024 > 49151 [SYN] Seq=0 Win=8192 Len=0 MSS=1460 WS=0 TSV=194 TSER=0
0000 f1 f1 f1 00 03 09 ab ab ab 60 10 89 08 00 45 00
0010 00 3c 00 68 40 00 40 06 6f 35 0a 0d 5b 02 0a 0d
0020 5c 03 04 00 bf ff 7d b3 81 44 00 00 00 00 a0 02
0030 20 00 9c 2f 00 00 02 04 05 b4 01 03 03 00 01 01
0040 08 0a 00 00 00 c2 00 00 00 00
64 2009-06-29 13:07:49.685375 10.13.92.3 10.13.91.2 TCP 49151 > 1024 [SYN, ACK] Seq=0 Ack=1 Win=1460 Len=0
0000 ab ab ab 60 10 89 f1 f1 f1 00 03 09 08 00 45 00
0010 00 28 00 02 00 00 64 06 8b af 0a 0d 5c 03 0a 0d
0020 5b 02 bf ff 04 00 d4 ff ff ff 7d b3 81 45 50 12
0030 05 b4 47 07 00 00 00 00 00 00 00 00
65 2009-06-29 13:07:49.685549 10.13.91.2 10.13.92.3 TCP 1024 > 49151 [RST] Seq=1 Win=0 Len=0
0000 f1 f1 f1 00 03 09 ab ab ab 60 10 89 08 00 45 00
0010 00 28 00 6a 00 00 40 06 af 47 0a 0d 5b 02 0a 0d
0020 5c 03 04 00 bf ff 7d b3 81 45 00 00 00 00 50 04
0030 00 00 21 c9 00 00 00 00 00 00 00 00

If both of you are using standard RTOS implementations, it is unlikely the TCP stack has a problem. Or, did you say the TCP is locally implemented?
If his client sends a SYN properly, and you can reply with a SYN+ACK,
it would appear that either your SYN+ACK is not well formed
(but, I could not see anything wrong yet), or,
like you suspect, his TCP stack did not accept the SYN+ACK properly.
However, if these are standard implementations, that is unlikely.
So, what more can you do?
Since it is the TCP handshake we are checking, you can just make him connect to any other machine at your end that is listening on the desired port
This will check his implementation (its good if the 3-way completes).
You can check your TCP stack with a TELNET connect to the port from another local machine
This will check your implementation (good if 3-way completes).
If both these things are fine, we need to suspect the network path
For example, could there be some firewall not allowing the communication and actively sending a RST to you?

First of all, those aren't valid MAC addresses; a high-order byte & 0x1 means it's a multicast MAC. See http://en.wikipedia.org/wiki/MAC_address

If you're not using fancy stuff on your side like custom tcp stack or raw sockets, I'd suspect the "proprietary TCP interface".
Has this ever worked with that client?
Does it work with other clients?

Related

to read the packet data from a cap file

i need to read the payload information from a udp packet which is in pcap file.
tshark -x -c 10 -r traces/trace.pcap udp
0000 00 22 4d a8 1f 7f 80 3f 5d 08 52 35 08 00 45 00 ."M....?].R5..E.
0010 05 94 03 e6 40 00 40 11 3b 46 c0 a8 ba 6c c0 a8 ....#.#.;F...l..
0020 ba 6f d4 80 05 dc 05 80 4e 9e 00 00 80 25 1b 3a .o......N....%.:
0030 fd 9e 00 00 00 01 00 00 00 06 5b 29 c2 5b 78 ed ..........[).[x.
0040 00 00 1b 6f c2 5b 78 ed 00 00 1e c4 30 5b 00 00 ...o.[x.....0[..
0050 00 00 85 c7 09 00 00 00 00 00 01 00 00 00 06 00 ................
0060 00 00 c0 07 1c 00 00 00 00 00 c0 07 3c 00 00 00 ............<...
0070 00 00 c0 07 3c 00 00 00 00 00 78 78 78 78 78 78 ....<.....xxxxxx
i get the above output from tshark but the payload consists of packet arrival and departure time from a udp server how i need to retrieve arrival and departure time from above information.
To print both the packet as hex (-x) and the summary with the timestamp in tshark add the -P option. To get the timestamp as absolute instead of relative value add the -ta option. For more information see the documentation.

openssl s_client only works with -tls1 switch to connect

We have some legacy systems that are still only support tls1 (there are plans to move off this soon, but not soon enough).
In order to connect to our new system, I have enabled tls1 connections. However, when i run a command like:
openssl s_client -connect host:port i get a failure to connect. When adding the -debug switch to see why i see the following:
CONNECTED(00000004)
write to 0x8000d02160 [0x8000d64000] (139 bytes => 139 (0x8B))
0000 - 80 89 01 03 01 00 60 00-00 00 20 00 00 39 00 00 ......`... ..9..
0010 - 38 00 00 35 00 00 88 00-00 87 00 00 84 00 00 16 8..5............
0020 - 00 00 13 00 00 0a 07 00-c0 00 00 33 00 00 32 00 ...........3..2.
0030 - 00 2f 00 00 45 00 00 44-00 00 41 03 00 80 00 00 ./..E..D..A.....
0040 - 05 00 00 04 01 00 80 00-00 15 00 00 12 00 00 09 ................
0050 - 06 00 40 00 00 14 00 00-11 00 00 08 00 00 06 04 ..#.............
0060 - 00 80 00 00 03 02 00 80-00 00 ff 29 c2 dd fb 71 ...........)...q
0070 - 5b 62 90 9e 5b b7 e7 5f-2e 67 9f a2 d2 01 eb bd [b..[.._.g......
0080 - 7f 16 28 2a 66 eb 37 78-92 d7 80 ..(*f.7x...
read from 0x8000d02160 [0x8000d6a000] (7 bytes => 0 (0x0))
59659:error:140790E5:SSL routines:SSL23_WRITE:ssl handshake failure:/home/src/secure/lib/libssl/../../../crypto/openssl/ssl/s23_lib.c:182:
but, when i add the -tls1 switch i get connected as expected. I am confused why this is happening. Shouldn't openssl try all acceptable methods when connecting ?
0000 - 80 89 01 03 01 ...
This is a SSLv2 compatible ClientHello (0x01) announcing support for TLS version 1.0 (0x0301). My guess is that the server does not understand a SSLv2 compatible handshake (long obsolete) but expects a proper TLS handshake which you can get with the -tls1 option.
Given that your openssl s_client does this SSLv2 compatible handshake by default and that it only supports TLS 1.0 and not better (since this is the largest it is announcing by default) suggests that you are using an old and unsupported version of OpenSSL, i.e. 0.9.8 or 1.0.0.
Shouldn't openssl try all acceptable methods when connecting ?
That's not how SSL/TLS works. There is not trying of various methods. Instead the client announces the best it can do (TLS 1.0 in your case) and the server picks a protocol version equal or lower to the version supported by the client, in the hope that the client will accept this.

Kafka SSL(Connection with node Disconnected)

I have configured SSL for kafka and I am having following two scenarios :-
A) When I configure the SSL listener on internal IP address (192.168.x.x), It works fine,
B) When I configure the SSL listener on eth1 IP address(10.x.x.x) because that is the only IP of broker my client can reach, It doesnt work but throw following error :-
[2017-04-07 09:04:31,566] DEBUG Completed connection to node -1 (org.apache.kafka.clients.NetworkClient)
[2017-04-07 09:06:17,860] DEBUG Connection with devmoomsg1/10.228.175.83 disconnected (org.apache.kafka.common.network.Selector)
Adding more, When I try to check with the openssl whether the firewall is blockiing the ssl traffic I get output fine.
OpenSSL> s_client -host x.x.x.x -port 443 -debug
CONNECTED(00000003)
write to 0x1500040 [0x1548fa0] (249 bytes => 249 (0xF9))
0000 - 16 03 01 00 f4 01 00 00-f0 03 03 58 e7 49 9b 1b ...........X.I..
0010 - 60 ba 05 65 af 2e d3 2f-18 61 2a 87 ad b6 82 c6 `..e.../.a*.....
0020 - cd 9b d5 16 89 07 dd 65-38 1d 53 00 00 84 c0 30 .......e8.S....0
0030 - c0 2c c0 28 c0 24 c0 13-c0 0a 00 a3 00 9f 00 6b .,.(.$.........k
0040 - 00 6a 00 39 00 38 00 99-00 87 c0 32 c0 2e c0 2a .j.9.8.....2...*
...
0080 - 00 44 00 16 00 13 c0 31-c0 2d c0 29 c0 25 c0 0e .D.....1.-.).%..
0090 - c0 04 c0 0d c0 03 00 9c-00 3c 00 2f 00 96 00 41 .........<./...A
....
00e0 - 04 01 04 02 04 03 03 01-03 02 03 03 02 01 02 02 ................
00f0 - 02 03 01 01 00 0f 00 01-01 .........

OpenSSL ClientHello fails in latest version

I need to connect to an SSL server. When I connect using OpenSSL 1.0.1r it works fine:
CONNECTED(00000304)
write to 0x6e13e8 [0x730a98] (297 bytes => 297 (0x129))
0000 - 16 03 01 01 24 01 00 01-20 03 03 16 f2 71 5f 26 ....$... ....q_&
0010 - a5 9b 64 cb 8f 0b 27 65-8d a3 54 e6 de a5 18 7a ..d...'e..T....z
0020 - 3c 5a e4 08 ab ff 6a 92-d7 45 f3 00 00 8a c0 30 <Z....j..E.....0
0030 - c0 2c c0 28 c0 24 c0 14-c0 0a 00 a3 00 9f 00 6b .,.(.$.........k
0040 - 00 6a 00 39 00 38 00 88-00 87 c0 32 c0 2e c0 2a .j.9.8.....2...*
0050 - c0 26 c0 0f c0 05 00 9d-00 3d 00 35 00 84 c0 2f .&.......=.5.../
0060 - c0 2b c0 27 c0 23 c0 13-c0 09 00 a2 00 9e 00 67 .+.'.#.........g
0070 - 00 40 00 33 00 32 00 9a-00 99 00 45 00 44 c0 31 .#.3.2.....E.D.1
0080 - c0 2d c0 29 c0 25 c0 0e-c0 04 00 9c 00 3c 00 2f .-.).%.......<./
0090 - 00 96 00 41 00 07 c0 11-c0 07 c0 0c c0 02 00 05 ...A............
00a0 - 00 04 c0 12 c0 08 00 16-00 13 c0 0d c0 03 00 0a ................
00b0 - 00 15 00 12 00 09 00 ff-01 00 00 6d 00 0b 00 04 ...........m....
00c0 - 03 00 01 02 00 0a 00 34-00 32 00 0e 00 0d 00 19 .......4.2......
00d0 - 00 0b 00 0c 00 18 00 09-00 0a 00 16 00 17 00 08 ................
00e0 - 00 06 00 07 00 14 00 15-00 04 00 05 00 12 00 13 ................
00f0 - 00 01 00 02 00 03 00 0f-00 10 00 11 00 23 00 00 .............#..
0100 - 00 0d 00 20 00 1e 06 01-06 02 06 03 05 01 05 02 ... ............
0110 - 05 03 04 01 04 02 04 03-03 01 03 02 03 03 02 01 ................
0120 - 02 02 02 03 00 0f 00 01-01 .........
>>> TLS 1.2 Handshake [length 0124], ClientHello
01 00 01 20 03 03 16 f2 71 5f 26 a5 9b 64 cb 8f
0b 27 65 8d a3 54 e6 de a5 18 7a 3c 5a e4 08 ab
ff 6a 92 d7 45 f3 00 00 8a c0 30 c0 2c c0 28 c0
24 c0 14 c0 0a 00 a3 00 9f 00 6b 00 6a 00 39 00
38 00 88 00 87 c0 32 c0 2e c0 2a c0 26 c0 0f c0
05 00 9d 00 3d 00 35 00 84 c0 2f c0 2b c0 27 c0
23 c0 13 c0 09 00 a2 00 9e 00 67 00 40 00 33 00
32 00 9a 00 99 00 45 00 44 c0 31 c0 2d c0 29 c0
25 c0 0e c0 04 00 9c 00 3c 00 2f 00 96 00 41 00
07 c0 11 c0 07 c0 0c c0 02 00 05 00 04 c0 12 c0
08 00 16 00 13 c0 0d c0 03 00 0a 00 15 00 12 00
09 00 ff 01 00 00 6d 00 0b 00 04 03 00 01 02 00
0a 00 34 00 32 00 0e 00 0d 00 19 00 0b 00 0c 00
18 00 09 00 0a 00 16 00 17 00 08 00 06 00 07 00
14 00 15 00 04 00 05 00 12 00 13 00 01 00 02 00
03 00 0f 00 10 00 11 00 23 00 00 00 0d 00 20 00
1e 06 01 06 02 06 03 05 01 05 02 05 03 04 01 04
02 04 03 03 01 03 02 03 03 02 01 02 02 02 03 00
0f 00 01 01
read from 0x6e13e8 [0x735ff8] (7 bytes => 7 (0x7))
0000 - 16 03 03 00 57 02 ....W.
0007 - <SPACES/NULS>
read from 0x6e13e8 [0x736002] (85 bytes => 85 (0x55))
0000 - 00 53 03 03 56 bb 11 21-1a ac 49 84 2d be 94 ad .S..V..!..I.-...
0010 - a0 c4 57 46 bc 70 d0 84-95 ce 96 c6 8c 92 07 2e ..WF.p..........
0020 - 4e 13 d6 f3 20 aa d7 86-ca 48 5e 01 a1 8c d3 f1 N... ....H^.....
0030 - d7 74 f9 2c 84 48 7d c1-95 6d 22 81 ff 53 ab d3 .t.,.H}..m"..S..
0040 - 0c 89 81 7d a2 00 3d 00-00 0b 00 0b 00 02 01 00 ...}..=.........
0050 - ff 01 00 01 ....
0055 - <SPACES/NULS>
<<< TLS 1.2 Handshake [length 0057], ServerHello
02 00 00 53 03 03 56 bb 11 21 1a ac 49 84 2d be
94 ad a0 c4 57 46 bc 70 d0 84 95 ce 96 c6 8c 92
07 2e 4e 13 d6 f3 20 aa d7 86 ca 48 5e 01 a1 8c
d3 f1 d7 74 f9 2c 84 48 7d c1 95 6d 22 81 ff 53
ab d3 0c 89 81 7d a2 00 3d 00 00 0b 00 0b 00 02
01 00 ff 01 00 01 00
read from 0x6e13e8 [0x735ffb] (5 bytes => 5 (0x5))
// etc.
However when I connect using OpenSSL 1.0.2f, the server closes the connection immediately:
CONNECTED(00000300)
write to 0x7812d0 [0x7ceef0] (317 bytes => 317 (0x13D))
0000 - 16 03 01 01 38 01 00 01-34 03 03 45 1c 09 c2 2e ....8...4..E....
0010 - 46 06 85 a1 01 fd 0a 2c-bb 6f 15 10 42 74 b3 bf F......,.o..Bt..
0020 - 9f 2e 5c 00 9f f2 93 8e-c0 18 9c 00 00 b6 c0 30 ..\............0
0030 - c0 2c c0 28 c0 24 c0 14-c0 0a 00 a5 00 a3 00 a1 .,.(.$..........
0040 - 00 9f 00 6b 00 6a 00 69-00 68 00 39 00 38 00 37 ...k.j.i.h.9.8.7
0050 - 00 36 00 88 00 87 00 86-00 85 c0 32 c0 2e c0 2a .6.........2...*
0060 - c0 26 c0 0f c0 05 00 9d-00 3d 00 35 00 84 c0 2f .&.......=.5.../
0070 - c0 2b c0 27 c0 23 c0 13-c0 09 00 a4 00 a2 00 a0 .+.'.#..........
0080 - 00 9e 00 67 00 40 00 3f-00 3e 00 33 00 32 00 31 ...g.#.?.>.3.2.1
0090 - 00 30 00 9a 00 99 00 98-00 97 00 45 00 44 00 43 .0.........E.D.C
00a0 - 00 42 c0 31 c0 2d c0 29-c0 25 c0 0e c0 04 00 9c .B.1.-.).%......
00b0 - 00 3c 00 2f 00 96 00 41-00 07 c0 11 c0 07 c0 0c .<./...A........
00c0 - c0 02 00 05 00 04 c0 12-c0 08 00 16 00 13 00 10 ................
00d0 - 00 0d c0 0d c0 03 00 0a-00 15 00 12 00 0f 00 0c ................
00e0 - 00 09 00 ff 01 00 00 55-00 0b 00 04 03 00 01 02 .......U........
00f0 - 00 0a 00 1c 00 1a 00 17-00 19 00 1c 00 1b 00 18 ................
0100 - 00 1a 00 16 00 0e 00 0d-00 0b 00 0c 00 09 00 0a ................
0110 - 00 23 00 00 00 0d 00 20-00 1e 06 01 06 02 06 03 .#..... ........
0120 - 05 01 05 02 05 03 04 01-04 02 04 03 03 01 03 02 ................
0130 - 03 03 02 01 02 02 02 03-00 0f 00 01 01 .............
>>> TLS 1.2 [length 0005]
16 03 01 01 38
>>> TLS 1.2 Handshake [length 0138], ClientHello
01 00 01 34 03 03 45 1c 09 c2 2e 46 06 85 a1 01
fd 0a 2c bb 6f 15 10 42 74 b3 bf 9f 2e 5c 00 9f
f2 93 8e c0 18 9c 00 00 b6 c0 30 c0 2c c0 28 c0
24 c0 14 c0 0a 00 a5 00 a3 00 a1 00 9f 00 6b 00
6a 00 69 00 68 00 39 00 38 00 37 00 36 00 88 00
87 00 86 00 85 c0 32 c0 2e c0 2a c0 26 c0 0f c0
05 00 9d 00 3d 00 35 00 84 c0 2f c0 2b c0 27 c0
23 c0 13 c0 09 00 a4 00 a2 00 a0 00 9e 00 67 00
40 00 3f 00 3e 00 33 00 32 00 31 00 30 00 9a 00
99 00 98 00 97 00 45 00 44 00 43 00 42 c0 31 c0
2d c0 29 c0 25 c0 0e c0 04 00 9c 00 3c 00 2f 00
96 00 41 00 07 c0 11 c0 07 c0 0c c0 02 00 05 00
04 c0 12 c0 08 00 16 00 13 00 10 00 0d c0 0d c0
03 00 0a 00 15 00 12 00 0f 00 0c 00 09 00 ff 01
00 00 55 00 0b 00 04 03 00 01 02 00 0a 00 1c 00
1a 00 17 00 19 00 1c 00 1b 00 18 00 1a 00 16 00
0e 00 0d 00 0b 00 0c 00 09 00 0a 00 23 00 00 00
0d 00 20 00 1e 06 01 06 02 06 03 05 01 05 02 05
03 04 01 04 02 04 03 03 01 03 02 03 03 02 01 02
02 02 03 00 0f 00 01 01
read from 0x7812d0 [0x7d4450] (7 bytes => 0 (0x0))
10124:error:140790E5:SSL routines:ssl23_write:ssl handshake failure:.\ssl\s23_lib.c:177:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 317 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : 0000
Session-ID:
Session-ID-ctx:
Master-Key:
Key-Arg : None
PSK identity: None
PSK identity hint: None
SRP username: None
Start Time: 1455100247
Timeout : 300 (sec)
Verify return code: 0 (ok)
---
I don't suppose anyone speaks ClientHello sufficiently well to tell me what the difference is, and how to get OpenSSL 1.0.2f to behave like the old version (even if it is insecure; I don't control the server).
I checked both traces with Wireshark, and the only significant differences I can see are that OpenSSL 1.0.2's ClientHello packet is identified as "SSL" by Wireshark, and the record layer is SSL:
Whereas OpenSSL 1.0.1 is identified as TLS 1.2. It also has fewer cypher suites (I guess they removed some insecure ones?).
I've tried the following combinations, and these are how Wireshark labels them in the "Protocol" column:
1.0.1: <no options>=TLSv1.2; -ssl2=SSLv2; -ssl3=SSLv3; -tls1_2=TLSv1.2
1.0.2: <no options>=SSL; -ssl2=SSLv2; -ssl3=SSLv3; -tls1_2=SSL
Any ideas about:
Why Wireshark decodes 1.0.1 and 1.0.2 differently?
Why the connection is failing with 1.0.2?
How I can get OpenSSL 1.0.2 to behave like 1.0.1?
Why Wireshark decodes 1.0.1 and 1.0.2 differently?
See SSL Record Layer vs SSLv3 Record Layer on the Wireshark Q&A bulletin board and Secure Socket Layer (SSL) on the Wireshark wiki.
Why the connection is failing with 1.0.2?
Looks like a buggy server that's rejecting a record layer that's not an early one, like one used in SSLv3.
The record layer is just that... It specifies the version of the record layer, which is simply the framing of SSL/TLS protocol messages. It is not a MIN-TLS-VERSION as many people think.
The TLS protocol version is just that... It specifies the version of the SSL/TLS protocol. It is not a MAX-TLS-VERSION as many people think.
How I can get OpenSSL 1.0.2 to behave like 1.0.1?
Use the following in your client, but its not exactly the same. The OpenSSL client will do the right thing and select TLS 1.2 if its available:
/* Uses the early record layer for downlevel servers */
const SSL_METHOD* method = SSLv23_method();
if(NULL == method) handleFailure();
ctx = SSL_CTX_new(method);
if(ctx == NULL) handleFailure();
/* Cannot fail ??? */
const long flags = SSL_OP_NO_SSLv2 | SSL_OP_NO_SSLv3 | SSL_OP_NO_TLSv1 | SSL_OP_NO_TLSv1_1 | SSL_OP_NO_COMPRESSION;
SSL_CTX_set_options(ctx, flags);
I checked OpenSSL 1.0.2's ssl/ssl.h, and both SSL_OP_NO_TLSv1 and SSL_OP_NO_TLSv1_1 are available.
There's a different way to do it for OpenSSL Master (a.k.a, OpenSSL 1.1.0 and above); see Working around servers requiring SSL 2/3 record layer, and using TLS 1.2?
... [OpenSSL 1.0.2] also has fewer cipher suites (I guess they removed some insecure ones?).
You should not leave cipher suites to chance. You should do something like the following:
const char* const PREFERRED_CIPHERS = "HIGH:!aNULL:!kRSA:!PSK:!SRP:!MD5:!RC4";
res = SSL_set_cipher_list(ssl, PREFERRED_CIPHERS);
if(res != 1) handleFailure();
The !PSK and !SRP simply removes cipher suites that are not usually used. !MD5 and !RC4 are removed for servers to help avoid the Obsolete cryptography warning from Browser.
You could even do the following:
const char* const PREFERRED_CIPHERS =
"ECDHE-ECDSA-AES256-GCM-SHA384:"
"ECDHE-RSA-AES256-GCM-SHA384:"
"ECDHE-ECDSA-AES128-GCM-SHA256:"
"ECDHE-RSA-AES128-GCM-SHA256:"
"DHE-DSS-AES256-GCM-SHA384:"
"DHE-RSA-AES256-GCM-SHA384:"
"DHE-DSS-AES128-GCM-SHA256:"
"DHE-RSA-AES128-GCM-SHA256";
res = SSL_set_cipher_list(ssl, PREFERRED_CIPHERS);
if(res != 1) handleFailure();
Be sure to always offer a AES/GCM cipher suite because as server configurations move to a TLS 1.2-only configuration, that's the cipher suite they usually choose.
Also, each cipher suite takes up to bytes in the ClientHello, and you want to minimize the number of them. You want to minimize them because some older Proxies and Interception boxes use a fixed size buffer for the client's ClientHello, and they can't handle the proliferation of cipher suite options available in TLS 1.2. The older boxes include F5 and Ironport middleware.
Why Wireshark decodes 1.0.1 and 1.0.2 differently?
No idea.
Why the connection is failing with 1.0.2?
Hard to tell but I've seen several servers having problems with some ciphers added to OpenSSL 1.0.2 or that too much ciphers were given or similar. I've seen similar problems with recent LibreSSL versions. In this case the servers have obviously broken TLS stacks.
How I can get OpenSSL 1.0.2 to behave like 1.0.1?
You can try to reduce the cipher set so that it only uses the same ciphers as OpenSSL 1.0.1. Notable I've seen servers croak ChaCha20-Poly1305 was inside the cipher set, probably they did not understand what kind of cipher this is.

How can I check INITIALIZE UPDATE and EXTERNAL AUTHENTICATE correctness?

I sent 80 50 00 00 08 00 00 00 00 00 00 00 00 [INITILIZE UPDATE Command] via opensc-tool to my java card and received 00 00 11 60 01 00 8A 79 0A F9 FF 02 00 11 79 11 36 5D 71 00 A5 A5 EC 63 BB DC 05 CC [Init Response] as its response from the card.
As you see:
In the command,I send 00 00 00 00 00 00 00 00 as Host Challenge, And in the response :
00 00 11 60 01 00 8A 79 0A F9 = Key diversification data
FF 02 = Key information
00 11 79 11 36 5D 71 00 = Card challenge
A5 A5 EC 63 BB DC 05 CC = Card cryptogram
Now I want to check myself,if the card cryptogram is OK or not. How I can do it? for example I encrypt 00 00 00 00 00 00 00 00 in this site under a 3DES cryptography algorithm [with keys of my card = 4041...4F], but the output is not equal with card cryptogram that I wrote above. Why?
And the next question is, if I want to send EXTERNAL AUTHENTICATION command to the card, what is its data field (after the above INITILIZE UPDATE)?
Update:
This is GPJ output :
C:\Users\ghasemi\Desktop\gpj-20120310>GPJ
C:\Users\ghasemi\Desktop\gpj-20120310>java -jar gpj.jar
Found terminals: [PC/SC terminal ACS CCID USB Reader 0]
Found card in terminal: ACS CCID USB Reader 0
ATR: 3B 68 00 00 00 73 C8 40 12 00 90 00
.
.
.
DEBUG: Command APDU: 00 A4 04 00 08 A0 00 00 00 03 00 00 00
DEBUG: Response APDU: 6F 10 84 08 A0 00 00 00 03 00 00 00 A5 04 9F 65 01 FF 90 00
Successfully selected Security Domain OP201a A0 00 00 00 03 00 00 00
DEBUG: Command APDU: 80 50 00 00 08 7F 41 A9 E7 19 37 83 FA
DEBUG: Response APDU: 00 00 11 60 01 00 8A 79 0A F9 FF 02 00 1B 9B 95 B9 5E 5E BC BA 51 34 84 D9 C1 B9 6E 90 00
DEBUG: Command APDU: 84 82 00 00 10 13 3B 4E C5 2C 9E D8 24 50 71 83 3A 78 AE 75 23
DEBUG: Response APDU: 90 00
DEBUG: Command APDU: 84 82 00 00 08 13 3B 4E C5 2C 9E D8 24
DEBUG: Response APDU: 90 00
C:\Users\ghasemi\Desktop\gpj-20120310>
So :
Host_Challenge :: 7F41A9E7193783FA
Diversification_Data :: 0000116001008A790AF9
Key_Information :: FF02
Sequence_Counter :: 001B
Card_Challenge :: 9B95B95E5EBC
Card_Cryptogram :: BA513484D9C1B96E
Host_Cryptogram[16,24] = 13 3B 4E C5 2C 9E D8 24
Now,lets make our Host_Cryptogram Manually :
Derivation_data=derivation_const_ENC|sequence_counter|0000 0000 0000 0000 0000 0000
Derivation_Data = 0182001B000000000000000000000000
k_ENC :: 404142434445464748494A4B4C4D4E4F
IV = 00 00 00 00 00 00 00 00
S_ENC = encrypt(TDES_CBC, K_ENC, IV, derivation_data)
So :
I used http://tripledes.online-domain-tools.com/ and its output for above values was :
S_ENC = 448b0a5967ca246d058703ff0c694f15
And :
Padding_DES = 80 00 00 00 00 00 00 00
Host_auth_data = sequence_counter | card_challenge | host_challenge | padding_DES
IV = Card_Cryptogram :: BA513484D9C1B96E
host_cryptogram = encrypt(TDES_CBC, S_ENC, IV, host_auth_data)
So :
Host_Authentication_Data : 001B9B95B95E5EBC7F41A9E7193783FA8000000000000000
Again, I used http://tripledes.online-domain-tools.com/
and :
Host_Cryptogram : 3587b531db71ac52392493c08cff189ce7b9061029c63b62
So :
Host_Cryptogram[16,24] = e7b9061029c63b62
Why these two way [manually and GPJ output] give us two host cryptogram?
From the INITIALIZE UPDATE command you send, you get
host_challenge = 00 00 00 00 00 00 00 00
In response to the INITIALIZE UPDATE command, you get
diversification_data = 00 00 11 60 01 00 8A 79 0A F9
key_information = FF 02
sequence_counter = 00 11
card_challenge = 79 11 36 5D 71 00
card_cryptogram = A5 A5 EC 63 BB DC 05 CC
The key information indicates SCP02 (02). The key diversification data may be used to derive the card-specific K_ENC. Lets assume we have a K_ENC like this
K_ENC = 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
We can then derive the session encryption key like this
derivation_const_ENC = 01 82
derivation_data = derivation_const_ENC | sequence_counter | 00 00 00 00 00 00 00 00 00 00 00 00
IV = 00 00 00 00 00 00 00 00
S_ENC = encrypt(TDES_CBC, K_ENC, IV, derivation_data)
Next, we can assemble the authentication data used to calculate the host cryptogram:
padding_DES = 80 00 00 00 00 00 00 00
host_auth_data = sequence_counter | card_challenge | host_challenge | padding_DES
Then we can use the session encryption key to encrypt the authentication data:
IV = 00 00 00 00 00 00 00 00
host_cryptogram = encrypt(TDES_CBC, S_ENC, IV, host_auth_data)
The last 8 bytes of the encrypted authentication data are the actual host cryptogram that we would send to the card:
EXTERNAL_AUTHENTICATE_data = host_cryptogram[16, 24]
Now we can assemble the EXTERNAL AUTHENTICATE command:
EXTERNAL_AUTHENTICATE = 84 82 03 00 08 | EXTERNAL_AUTHENTICATE_data
We can then calculate the S_MAC key (analoguous to getting the S_ENC above) and the MAC over that command and append it to the command data to get the full EXTERNAL AUTHENTICATE command that can be sent to the card:
EXTERNAL_AUTHENTICATE = 84 82 03 00 10 | EXTERNAL_AUTHENTICATE_data | MAC
Update
Using http://tripledes.online-domain-tools.com/ to reproduce the results of GPJ
Your K_ENC is 404142434445464748494A4B4C4D4E4F. The online tools does not properly support 2-key-3DES, so you have to convert the key into its 3-key form first:
K_ENC = 404142434445464748494A4B4C4D4E4F4041424344454647
Use this key and a zero IV to encrypt the derivation data (0182001B000000000000000000000000). You get
S_ENC = fb063cc2e17b979b10e22f82110234b4
In 3-key notation, this is
S_ENC = fb063cc2e17b979b10e22f82110234b4fb063cc2e17b979b
Use this key and a zero IV to encrypt the host authentication data (001b9b95b95e5ebc7f41a9e7193783fa8000000000000000):
HOST_CRYPTOGRAM = 773e790c91acce3167d99f92c60e2afd133b4ec52c9ed824