I am trying to find out if it is possible to confirm or check that a client certificate is being sent (from the client-side) when a 2-way SSL connection is being established, using tcpdump (and if so, how?). I am trying to do this because we need to validate that a 2-way SSL connection (vs. just a 1-way SSL connection) is actually being established.
In our configuration, the "client side" is an Oracle OAM Apache webgate, and the "server side" is an OAM server.
Connection is from the webgate ==> OAM server, listening on (for example) port 5575.
The connection is established when the webgate processes a request the 1st time, and the connection stays up for awhile after that, so the way I have been testing this is:
Start up the Apache (the webgate is an Apache module). At this point there is no connection to the OAM server port 5575.
Send a request (using a browser) to the Apache. This causes the webgate to establish a (supposedly 2-way SSL) connection to the OAM server port 5575.
I have been trying various tcpdump command lines (run on the client/webgate machine) to try to show/capture the webgate's client cert, but so far, I have not been able to see any indication that the webgate is sending a client cert.
I am not very familiar with tcpdump, so I was hoping that someone could suggest how to do this.
Thanks!
Jim
EDIT: I hope that this is ok to add on here. FYI, I have been unable to directly confirm whether 2-way SSL is occurring. However, I was able to do the following:
I did some tcpdump captures and read that into ssldump:
New TCP connection #2: 192.168.aa.bb(37256) <-> 172.31.xx.yy(5575)
0.0011 (0.0011) C>S
---------------------------------------------------------------
4e 4d 50 04 01 NMP..
---------------------------------------------------------------
0.0034 (0.0022) S>C
---------------------------------------------------------------
4e 4d 50 05 01 NMP..
---------------------------------------------------------------
0.0035 (0.0001) C>S
---------------------------------------------------------------
00 00 00 34 00 00 00 00 00 0d 70 72 6f 74 6f 63 ...4......protoc
6f 6c 3d 4e 41 50 20 76 65 72 73 69 6f 6e 3d 34 ol=NAP version=4
20 6f 6c 64 65 73 74 3d 31 20 63 6d 3d 6a 6c 74 oldest=1 cm=jlt
65 73 74 31 est1
---------------------------------------------------------------
0.0052 (0.0017) S>C
---------------------------------------------------------------
00 00 00 3e 00 00 00 00 00 0d 70 72 6f 74 6f 63 ...>......protoc
6f 6c 3d 4e 41 50 20 76 65 72 73 69 6f 6e 3d 35 ol=NAP version=5
20 6f 6c 64 65 73 74 3d 31 20 63 6c 69 65 6e 74 oldest=1 client
5f 6e 61 70 5f 76 65 72 73 69 6f 6e 3d 34 _nap_version=4
---------------------------------------------------------------
0.0054 (0.0001) C>S
---------------------------------------------------------------
00 00 00 12 00 00 00 00 01 00 73 74 73 3d 63 65 ..........sts=ce
72 74 rt
---------------------------------------------------------------
0.0097 (0.0043) S>C
---------------------------------------------------------------
00 00 00 12 00 00 00 00 01 00 73 74 73 3d 63 65 ..........sts=ce
72 74 rt
---------------------------------------------------------------
2 1 0.0106 (0.0008) C>S V3.3(81) Handshake
ClientHello
Version 3.3
random[32]=
61 08 bc db 2a 3d e1 ac 7e fc e4 c8 cb 34 8b d4
65 f5 35 bc 31 89 d4 4b 99 5f 3b 22 f5 5a 67 bf
cipher suites
TLS_RSA_WITH_AES_256_CBC_SHA
TLS_RSA_WITH_3DES_EDE_CBC_SHA
TLS_EMPTY_RENEGOTIATION_INFO_SCSV
compression methods
NULL
extensions
signature_algorithms
ja3 string: 771,53-10-255,13,,
ja3 fingerprint: 9331064077bda87e0c2780d544d22b00
2 2 0.0128 (0.0022) S>C V3.3(2041) Handshake
ServerHello
Version 3.3
random[32]=
13 c8 0a 1c 90 28 eb 39 55 b0 2f 55 ba 33 b1 9b
7b 45 f6 be 20 33 13 fa 44 4f 57 4e 47 52 44 01
session_id[32]=
0d 07 2d cf c8 a3 90 27 22 70 6b 5c cf e0 1d 36
f1 6c af 88 59 5f 74 1a 9b 35 df e8 e6 f8 da 78
cipherSuite TLS_RSA_WITH_AES_256_CBC_SHA
compressionMethod NULL
extensions
renegotiation_info
ja3s string: 771,53,65281
ja3s fingerprint: 7b6819ed58e8d8415604b7dfcef92d55
Certificate
ServerHelloDone
2 3 0.0136 (0.0007) C>S V3.3(262) Handshake
ClientKeyExchange
EncryptedPreMasterSecret[256]=
31 aa 1d a9 63 5c ff f8 29 0b 24 5d 7e ee 98 ff
cf 25 94 2f 55 5d 48 91 c1 67 c4 ea 39 71 7e ce
3c a6 a7 18 2c 3d f5 a4 07 1f 54 1e 01 82 b3 7b
46 fe 46 90 25 79 b2 46 5e 03 c3 4f df 81 89 d7
.
.
.
b5 e8 44 d5 cd 92 02 82 f3 42 27 56 5d 23 5b c1
9a e8 8e ec 4b 35 e8 4e 57 01 cb bc 61 09 66 c0
5c 8d 20 47 b4 da ef 2d 25 71 c8 7f 73 3b 47 7a
43 04 0b 6d 27 b3 c3 af 17 89 af d9 04 c1 66 a1
2 4 0.0136 (0.0000) C>S V3.3(1) ChangeCipherSpec
2 5 0.0136 (0.0000) C>S V3.3(64) Handshake
2 6 0.0165 (0.0029) S>C V3.3(1) ChangeCipherSpec
2 7 0.0565 (0.0400) S>C V3.3(64) Handshake
2 8 0.0567 (0.0002) C>S V3.3(96) application_data
2 9 0.0589 (0.0021) S>C V3.3(160) application_data
2 10 0.0591 (0.0002) C>S V3.3(96) application_data
2 11 0.0615 (0.0023) S>C V3.3(144) application_data
2 12 0.0617 (0.0001) C>S V3.3(528) application_data
2 13 0.1017 (0.0400) C>S V3.3(80) application_data
2 14 0.1042 (0.0025) S>C V3.3(1296) application_data
2 15 0.1116 (0.0074) C>S V3.3(288) application_data
2 16 0.1145 (0.0028) S>C V3.3(352) application_data
2 17 0.1179 (0.0034) C>S V3.3(64) application_data
2 18 0.1205 (0.0026) S>C V3.3(336) application_data
2 19 4.6999 (4.5793) C>S V3.3(1664) application_data
2 20 4.7183 (0.0184) S>C V3.3(1296) application_data
Cleaned 5 remaining connection(s) from connection pool
Notice in the above that there is no CertificateRequest, etc.
I did some additional "openssl s_client" connection tests, and it appears that there is no "list of acceptable CAs" sent by the server-side during the SSL handshake.
So I was wondering if: Would the above 2 pieces of information seem to imply that there was no 2-way SSL connection being established?
I am trying to debug the SSL connection of a client I do not have control over.
A possibility I discovered is to use openssl as a server to check whether the SSL connection is correctly initiated, and if not - why.
The basic connection ends with
root#elk:~# openssl s_server -key sonoff.key -cert sonoff.crt -accept 5555 -www
Using default temp DH parameters
ACCEPT
bad gethostbyaddr
140612029986456:error:140780E5:SSL routines:ssl23_read:ssl handshake failure:s23_lib.c:137:
What does this error actually mean?
The searches I did point to various bugs, but they all gravitate around "incorrect certificates". The certificates I use are correct, although self-signed. What in the message (or the -debug version below) could suggest this? Specifically: could this be an indication of the device checking the certificate and aborting on a self-signed?
root#elk:~# openssl s_server -key sonoff.key -cert sonoff.crt -accept 5555 -www -debug
Using default temp DH parameters
ACCEPT
bad gethostbyaddr
read from 0xb2ca20 [0xb32110] (11 bytes => 11 (0xB))
0000 - 16 03 01 00 4f 01 00 00-4b 03 03 ....O...K..
read from 0xb2ca20 [0xb3211e] (73 bytes => 73 (0x49))
0000 - 1b da e3 8f 9a d2 af 66-81 b5 62 95 7e 44 fc 3b .......f..b.~D.;
0010 - 75 48 90 da 51 03 f6 ad-33 81 ec 15 16 42 2b f2 uH..Q...3....B+.
0020 - 00 00 12 00 3d 00 35 00-3c 00 2f 00 b7 00 95 00 ....=.5.<./.....
0030 - b6 00 94 00 ff 01 00 00-10 00 0d 00 0c 00 0a 06 ................
0040 - 01 05 01 04 01 03 01 02-01 .........
write to 0xb2ca20 [0xb3bc00] (86 bytes => 86 (0x56))
0000 - 16 03 03 00 51 02 00 00-4d 03 03 2b ca 7c 43 47 ....Q...M..+.|CG
0010 - 06 ae 7e 1e b3 9c 71 7a-ca 5d 6d 98 54 4e 1c 5a ..~...qz.]m.TN.Z
0020 - b7 f9 38 2c 0d 3e de e0-62 a6 8a 20 2a 42 cf 9e ..8,.>..b.. *B..
0030 - ba 1e 8c a3 03 24 e1 34-9e 15 2e d8 90 77 73 74 .....$.4.....wst
0040 - 78 33 e6 45 83 34 ca 11-ae 97 4e 18 00 3d 00 00 x3.E.4....N..=..
0050 - 05 ff 01 00 01 .....
0056 - <SPACES/NULS>
write to 0xb2ca20 [0xb3bc00] (1298 bytes => 1298 (0x512))
0000 - 16 03 03 05 0d 0b 00 05-09 00 05 06 00 05 03 30 ...............0
0010 - 82 04 ff 30 82 02 e7 a0-03 02 01 02 02 09 00 df ...0............
0020 - 41 71 c6 48 ff eb 19 30-0d 06 09 2a 86 48 86 f7 Aq.H...0...*.H..
0030 - 0d 01 01 0b 05 00 30 16-31 14 30 12 06 03 55 04 ......0.1.0...U.
0040 - 03 0c 0b 65 78 61 6d 70-6c 65 2e 63 6f 6d 30 1e ...example.com0.
0050 - 17 0d 31 38 30 32 30 34-31 38 31 33 35 36 5a 17 ..180204181356Z.
0060 - 0d 32 38 30 32 30 32 31-38 31 33 35 36 5a 30 16 .280202181356Z0.
0070 - 31 14 30 12 06 03 55 04-03 0c 0b 65 78 61 6d 70 1.0...U....examp
0080 - 6c 65 2e 63 6f 6d 30 82-02 22 30 0d 06 09 2a 86 le.com0.."0...*.
0090 - 48 86 f7 0d 01 01 01 05-00 03 82 02 0f 00 30 82 H.............0.
00a0 - 02 0a 02 82 02 01 00 e0-22 b0 41 79 e5 60 92 2c ........".Ay.`.,
00b0 - c7 1c 97 84 f8 16 46 39-27 67 26 ad d7 c8 dc a4 ......F9'g&.....
00c0 - 18 2e 5a 1a aa f7 92 b0-71 8f f8 a5 e0 d9 52 d3 ..Z.....q.....R.
00d0 - 6a c9 e1 ae 7c ce f1 1c-92 99 dd 77 b7 f7 db b3 j...|......w....
00e0 - 2b f4 96 23 d4 d8 08 e8-e2 f2 15 7b 41 16 30 3d +..#.......{A.0=
00f0 - 21 cd 5f b0 3d 5a 0c ec-60 9b d8 78 99 32 c4 9f !._.=Z..`..x.2..
0100 - 3c 1b 50 d9 d4 cf f0 4b-58 ca c6 6f 56 f5 57 04 <.P....KX..oV.W.
0110 - c9 d5 60 56 d2 25 66 ee-fe f0 da 79 d6 f2 a5 67 ..`V.%f....y...g
0120 - 08 7a 19 e6 82 90 5c e0-a1 63 cf 4b a8 43 c6 d3 .z....\..c.K.C..
0130 - 08 a0 22 4a df be 42 22-8b c9 6f 81 d1 b0 d8 e0 .."J..B"..o.....
0140 - 24 79 b3 3a ad 10 af bd-7a 43 56 a6 87 ad d2 d4 $y.:....zCV.....
0150 - 96 cf 12 f9 9a 42 7a c7-63 07 f7 cc a9 a5 50 1e .....Bz.c.....P.
0160 - bf 4d 2d ae e3 74 b7 a7-66 ca ab 44 66 db f5 e2 .M-..t..f..Df...
0170 - 28 dd a8 6f 76 0a 5a d4-8d 42 c6 a0 16 c2 bd d9 (..ov.Z..B......
0180 - 25 38 74 dc bc d3 95 03-b9 13 39 c0 4a cd d9 c9 %8t.......9.J...
0190 - cb 34 bf 9e 48 1e e4 56-8f 47 9c 33 ed eb d6 db .4..H..V.G.3....
01a0 - 5e b7 46 85 d4 49 48 f3-24 ec 01 c3 98 74 50 f9 ^.F..IH.$....tP.
01b0 - 05 71 01 b5 96 f4 b5 bc-a3 57 78 3f c7 4f 10 1e .q.......Wx?.O..
01c0 - da 96 59 c0 6d c1 13 64-52 7b f8 d7 88 59 29 ba ..Y.m..dR{...Y).
01d0 - b7 5a ec a2 a7 1e 86 bb-ea 8b f8 03 e3 73 b3 6e .Z...........s.n
01e0 - 70 27 29 b6 92 31 4f 52-73 f0 62 91 3e 90 53 40 p')..1ORs.b.>.S#
01f0 - d0 00 e2 1d 80 5e 82 a9-5e fc 56 2f 76 d9 40 39 .....^..^.V/v.#9
0200 - 85 e3 4e 55 d2 a7 47 06-27 92 d9 ec dd 7e 7d c2 ..NU..G.'....~}.
0210 - 07 6d 73 9b 58 75 da ba-cd 65 e3 d9 3b da 78 4e .ms.Xu...e..;.xN
0220 - 96 81 c1 d1 2f f2 c0 65-e0 a2 12 33 1c eb 0c 2a ..../..e...3...*
0230 - 10 c6 ea 87 4c 7a a5 d7-c3 4c a8 45 06 8d ca a5 ....Lz...L.E....
0240 - 90 7d e0 57 be e5 17 f0-13 41 28 6e f6 f3 cb b3 .}.W.....A(n....
0250 - d4 9b fa b1 f9 30 a1 b4-2e a4 de 73 39 35 82 32 .....0.....s95.2
0260 - de 55 f9 70 ee ca 8a d2-62 64 fc fb d7 56 17 c7 .U.p....bd...V..
0270 - 9a 3c 4f 3d b1 4a 1d 37-fb ad 40 ef c4 14 cf 08 .<O=.J.7..#.....
0280 - 57 8e 68 5f 2b d4 3b cc-bb 23 9f 83 64 e3 22 39 W.h_+.;..#..d."9
0290 - 53 30 5c 4d 0f d8 4d 2a-5a 23 e6 ee 06 b6 38 5d S0\M..M*Z#....8]
02a0 - 83 ef 91 f8 25 a6 e3 02-03 01 00 01 a3 50 30 4e ....%........P0N
02b0 - 30 1d 06 03 55 1d 0e 04-16 04 14 bf 22 b8 2a ec 0...U.......".*.
02c0 - f6 61 a9 05 50 d8 6a f5-35 bd 32 3e 7c 9b d7 30 .a..P.j.5.2>|..0
02d0 - 1f 06 03 55 1d 23 04 18-30 16 80 14 bf 22 b8 2a ...U.#..0....".*
02e0 - ec f6 61 a9 05 50 d8 6a-f5 35 bd 32 3e 7c 9b d7 ..a..P.j.5.2>|..
02f0 - 30 0c 06 03 55 1d 13 04-05 30 03 01 01 ff 30 0d 0...U....0....0.
0300 - 06 09 2a 86 48 86 f7 0d-01 01 0b 05 00 03 82 02 ..*.H...........
0310 - 01 00 35 e5 1e a9 74 4f-78 59 c0 fd da a4 71 7a ..5...tOxY....qz
0320 - 4e 9d b7 13 42 5b d5 f3-95 1b 48 a6 91 c8 84 be N...B[....H.....
0330 - fc 67 ed 92 4e 2e f2 e5-6e 14 38 a5 af 51 7f 54 .g..N...n.8..Q.T
0340 - 11 5f ed 15 bd a6 d9 ef-9c ef ee db ed fb 39 45 ._............9E
0350 - e2 5e a9 53 66 73 40 98-27 38 4d a3 1d 5f eb d3 .^.Sfs#.'8M.._..
0360 - f8 33 01 3d ce 08 61 60-e1 ee 91 28 98 0a a2 17 .3.=..a`...(....
0370 - 28 17 39 c1 af 1e d2 f7-42 c6 5f ff a2 fa 77 2a (.9.....B._...w*
0380 - f0 4b ed 4f ea d4 55 b4-4d 02 63 4f 9f 12 ab 86 .K.O..U.M.cO....
0390 - 26 a6 24 26 7e 26 e2 55-89 6b 7d 52 3c 48 ad 4c &.$&~&.U.k}R<H.L
03a0 - 0c 76 64 7d a7 8f ff 9e-23 28 4c d5 79 85 ea 0c .vd}....#(L.y...
03b0 - ee 54 31 24 e2 66 53 4c-ae a6 a6 43 c5 e1 16 c7 .T1$.fSL...C....
03c0 - 48 b3 af 49 c9 f6 30 e6-4a da 2f 9b 1a 9d 5f 15 H..I..0.J./..._.
03d0 - ff ac 9e c7 d3 e4 85 55-1d d6 42 af 58 1e 57 d1 .......U..B.X.W.
03e0 - 84 4f 87 a7 43 10 e8 13-42 fd 49 fa c1 a2 ba 6a .O..C...B.I....j
03f0 - 30 a4 1e fb db be 99 16-a1 dc 1e 45 45 54 c3 89 0..........EET..
0400 - b8 b6 b8 74 90 2f 8b 8c-68 49 07 a2 7c 0d a6 d6 ...t./..hI..|...
0410 - 3c 80 e2 94 66 b0 83 1f-e3 29 02 4d 76 44 1e 65 <...f....).MvD.e
0420 - 57 7c 9e f2 01 0a 77 cd-c5 85 43 53 9e e7 56 21 W|....w...CS..V!
0430 - 4b 55 05 78 0e 6b 7a 14-44 69 d8 cb 06 de 65 9a KU.x.kz.Di....e.
0440 - 0a 8b 15 11 4e 21 c6 1d-be d4 6c 45 96 88 f2 04 ....N!....lE....
0450 - d5 ff 03 fd d3 24 7d 25-22 8b 22 37 99 4b 22 fa .....$}%"."7.K".
0460 - 8f 99 ec 71 88 58 8e 03-9f 76 19 a2 05 a6 3b bc ...q.X...v....;.
0470 - 87 d8 90 db c1 cc 98 f9-1f c5 00 96 e4 90 3d ad ..............=.
0480 - d0 fa a1 d5 b6 71 74 7d-0c a3 f2 79 5a e6 31 fb .....qt}...yZ.1.
0490 - de 65 41 46 39 78 bd d6-c1 b0 1a 44 16 a1 ed 27 .eAF9x.....D...'
04a0 - c6 e7 d6 ab f0 2f 0f 78-a6 78 01 2c ed 18 bb 47 ...../.x.x.,...G
04b0 - 34 54 ee 73 1b 99 83 ca-bf d7 07 65 40 03 f2 75 4T.s.......e#..u
04c0 - 98 d4 f3 22 6c 2f 77 1e-ec e7 1e bf 31 28 90 c4 ..."l/w.....1(..
04d0 - 0c 7a 37 cc 85 37 59 42-94 21 90 05 9e e0 60 51 .z7..7YB.!....`Q
04e0 - 0a 83 b8 29 45 37 06 5a-5e a3 b3 e0 08 cf 48 9b ...)E7.Z^.....H.
04f0 - 51 17 7b 4c fb 6d 06 e9-4f dd 45 42 42 83 c2 f6 Q.{L.m..O.EBB...
0500 - b3 70 29 86 08 c4 28 f7-b4 d6 1f a1 63 94 f7 75 .p)...(.....c..u
0510 - 8f ba ..
write to 0xb2ca20 [0xb3bc00] (9 bytes => 9 (0x9))
0000 - 16 03 03 00 04 0e ......
0009 - <SPACES/NULS>
read from 0xb2ca20 [0xb32113] (5 bytes => 0 (0x0))
140570414024344:error:140780E5:SSL routines:ssl23_read:ssl handshake failure:s23_lib.c:137:
EDIT: a K12 dump of the session:
+---------+---------------+----------+
18:19:07,529,418 ETHER
|0 |8a|4b|47|20|ce|e4|1e|33|c8|aa|1b|40|08|00|45|00|00|2c|05|fb|00|00|7f|06|16|28|0a|64|0a|71|0a|c8|00|0d|58|a7|15|b3|00|36|bf|c5|00|00|00|00|60|02|16|d0|33|57|00|00|02|04|05|b4|
+---------+---------------+----------+
18:19:07,529,450 ETHER
|0 |1e|33|c8|aa|1b|40|8a|4b|47|20|ce|e4|08|00|45|00|00|2c|00|00|40|00|40|06|1b|23|0a|c8|00|0d|0a|64|0a|71|15|b3|58|a7|62|61|d4|6d|00|36|bf|c6|60|12|72|10|1f|c8|00|00|02|04|05|b4|
+---------+---------------+----------+
18:19:07,668,343 ETHER
|0 |8a|4b|47|20|ce|e4|1e|33|c8|aa|1b|40|08|00|45|00|00|7c|05|fc|00|00|7f|06|15|d7|0a|64|0a|71|0a|c8|00|0d|58|a7|15|b3|00|36|bf|c6|62|61|d4|6e|50|18|16|d0|67|d0|00|00|16|03|01|00|4f|01|00|00|4b|03|03|6e|30|37|19|a3|c0|a4|d5|d1|3c|8b|ed|76|72|e2|9f|d5|5e|57|3e|9e|98|b7|33|a9|12|54|de|40|32|df|34|00|00|12|00|3d|00|35|00|3c|00|2f|00|b7|00|95|00|b6|00|94|00|ff|01|00|00|10|00|0d|00|0c|00|0a|06|01|05|01|04|01|03|01|02|01|
+---------+---------------+----------+
18:19:07,668,359 ETHER
|0 |1e|33|c8|aa|1b|40|8a|4b|47|20|ce|e4|08|00|45|00|00|28|39|19|40|00|40|06|e2|0d|0a|c8|00|0d|0a|64|0a|71|15|b3|58|a7|62|61|d4|6e|00|36|c0|1a|50|10|72|10|1f|c4|00|00|
+---------+---------------+----------+
18:19:07,668,617 ETHER
|0 |1e|33|c8|aa|1b|40|8a|4b|47|20|ce|e4|08|00|45|00|05|99|39|1a|40|00|40|06|dc|9b|0a|c8|00|0d|0a|64|0a|71|15|b3|58|a7|62|61|d4|6e|00|36|c0|1a|50|18|72|10|25|35|00|00|16|03|01|00|51|02|00|00|4d|03|01|e9|4e|66|d1|56|43|c2|f8|df|95|62|d2|a1|8f|99|85|39|d3|22|4e|92|64|85|75|47|8f|29|f5|15|11|8e|10|20|b7|07|bc|84|e7|15|1e|6b|53|ed|6c|d5|1c|f6|15|10|ab|03|94|db|15|19|47|b7|43|04|30|59|fb|f2|52|e5|00|35|00|00|05|ff|01|00|01|00|16|03|01|05|0d|0b|00|05|09|00|05|06|00|05|03|30|82|04|ff|30|82|02|e7|a0|03|02|01|02|02|09|00|df|41|71|c6|48|ff|eb|19|30|0d|06|09|2a|86|48|86|f7|0d|01|01|0b|05|00|30|16|31|14|30|12|06|03|55|04|03|0c|0b|65|78|61|6d|70|6c|65|2e|63|6f|6d|30|1e|17|0d|31|38|30|32|30|34|31|38|31|33|35|36|5a|17|0d|32|38|30|32|30|32|31|38|31|33|35|36|5a|30|16|31|14|30|12|06|03|55|04|03|0c|0b|65|78|61|6d|70|6c|65|2e|63|6f|6d|30|82|02|22|30|0d|06|09|2a|86|48|86|f7|0d|01|01|01|05|00|03|82|02|0f|00|30|82|02|0a|02|82|02|01|00|e0|22|b0|41|79|e5|60|92|2c|c7|1c|97|84|f8|16|46|39|27|67|26|ad|d7|c8|dc|a4|18|2e|5a|1a|aa|f7|92|b0|71|8f|f8|a5|e0|d9|52|d3|6a|c9|e1|ae|7c|ce|f1|1c|92|99|dd|77|b7|f7|db|b3|2b|f4|96|23|d4|d8|08|e8|e2|f2|15|7b|41|16|30|3d|21|cd|5f|b0|3d|5a|0c|ec|60|9b|d8|78|99|32|c4|9f|3c|1b|50|d9|d4|cf|f0|4b|58|ca|c6|6f|56|f5|57|04|c9|d5|60|56|d2|25|66|ee|fe|f0|da|79|d6|f2|a5|67|08|7a|19|e6|82|90|5c|e0|a1|63|cf|4b|a8|43|c6|d3|08|a0|22|4a|df|be|42|22|8b|c9|6f|81|d1|b0|d8|e0|24|79|b3|3a|ad|10|af|bd|7a|43|56|a6|87|ad|d2|d4|96|cf|12|f9|9a|42|7a|c7|63|07|f7|cc|a9|a5|50|1e|bf|4d|2d|ae|e3|74|b7|a7|66|ca|ab|44|66|db|f5|e2|28|dd|a8|6f|76|0a|5a|d4|8d|42|c6|a0|16|c2|bd|d9|25|38|74|dc|bc|d3|95|03|b9|13|39|c0|4a|cd|d9|c9|cb|34|bf|9e|48|1e|e4|56|8f|47|9c|33|ed|eb|d6|db|5e|b7|46|85|d4|49|48|f3|24|ec|01|c3|98|74|50|f9|05|71|01|b5|96|f4|b5|bc|a3|57|78|3f|c7|4f|10|1e|da|96|59|c0|6d|c1|13|64|52|7b|f8|d7|88|59|29|ba|b7|5a|ec|a2|a7|1e|86|bb|ea|8b|f8|03|e3|73|b3|6e|70|27|29|b6|92|31|4f|52|73|f0|62|91|3e|90|53|40|d0|00|e2|1d|80|5e|82|a9|5e|fc|56|2f|76|d9|40|39|85|e3|4e|55|d2|a7|47|06|27|92|d9|ec|dd|7e|7d|c2|07|6d|73|9b|58|75|da|ba|cd|65|e3|d9|3b|da|78|4e|96|81|c1|d1|2f|f2|c0|65|e0|a2|12|33|1c|eb|0c|2a|10|c6|ea|87|4c|7a|a5|d7|c3|4c|a8|45|06|8d|ca|a5|90|7d|e0|57|be|e5|17|f0|13|41|28|6e|f6|f3|cb|b3|d4|9b|fa|b1|f9|30|a1|b4|2e|a4|de|73|39|35|82|32|de|55|f9|70|ee|ca|8a|d2|62|64|fc|fb|d7|56|17|c7|9a|3c|4f|3d|b1|4a|1d|37|fb|ad|40|ef|c4|14|cf|08|57|8e|68|5f|2b|d4|3b|cc|bb|23|9f|83|64|e3|22|39|53|30|5c|4d|0f|d8|4d|2a|5a|23|e6|ee|06|b6|38|5d|83|ef|91|f8|25|a6|e3|02|03|01|00|01|a3|50|30|4e|30|1d|06|03|55|1d|0e|04|16|04|14|bf|22|b8|2a|ec|f6|61|a9|05|50|d8|6a|f5|35|bd|32|3e|7c|9b|d7|30|1f|06|03|55|1d|23|04|18|30|16|80|14|bf|22|b8|2a|ec|f6|61|a9|05|50|d8|6a|f5|35|bd|32|3e|7c|9b|d7|30|0c|06|03|55|1d|13|04|05|30|03|01|01|ff|30|0d|06|09|2a|86|48|86|f7|0d|01|01|0b|05|00|03|82|02|01|00|35|e5|1e|a9|74|4f|78|59|c0|fd|da|a4|71|7a|4e|9d|b7|13|42|5b|d5|f3|95|1b|48|a6|91|c8|84|be|fc|67|ed|92|4e|2e|f2|e5|6e|14|38|a5|af|51|7f|54|11|5f|ed|15|bd|a6|d9|ef|9c|ef|ee|db|ed|fb|39|45|e2|5e|a9|53|66|73|40|98|27|38|4d|a3|1d|5f|eb|d3|f8|33|01|3d|ce|08|61|60|e1|ee|91|28|98|0a|a2|17|28|17|39|c1|af|1e|d2|f7|42|c6|5f|ff|a2|fa|77|2a|f0|4b|ed|4f|ea|d4|55|b4|4d|02|63|4f|9f|12|ab|86|26|a6|24|26|7e|26|e2|55|89|6b|7d|52|3c|48|ad|4c|0c|76|64|7d|a7|8f|ff|9e|23|28|4c|d5|79|85|ea|0c|ee|54|31|24|e2|66|53|4c|ae|a6|a6|43|c5|e1|16|c7|48|b3|af|49|c9|f6|30|e6|4a|da|2f|9b|1a|9d|5f|15|ff|ac|9e|c7|d3|e4|85|55|1d|d6|42|af|58|1e|57|d1|84|4f|87|a7|43|10|e8|13|42|fd|49|fa|c1|a2|ba|6a|30|a4|1e|fb|db|be|99|16|a1|dc|1e|45|45|54|c3|89|b8|b6|b8|74|90|2f|8b|8c|68|49|07|a2|7c|0d|a6|d6|3c|80|e2|94|66|b0|83|1f|e3|29|02|4d|76|44|1e|65|57|7c|9e|f2|01|0a|77|cd|c5|85|43|53|9e|e7|56|21|4b|55|05|78|0e|6b|7a|14|44|69|d8|cb|06|de|65|9a|0a|8b|15|11|4e|21|c6|1d|be|d4|6c|45|96|88|f2|04|d5|ff|03|fd|d3|24|7d|25|22|8b|22|37|99|4b|22|fa|8f|99|ec|71|88|58|8e|03|9f|76|19|a2|05|a6|3b|bc|87|d8|90|db|c1|cc|98|f9|1f|c5|00|96|e4|90|3d|ad|d0|fa|a1|d5|b6|71|74|7d|0c|a3|f2|79|5a|e6|31|fb|de|65|41|46|39|78|bd|d6|c1|b0|1a|44|16|a1|ed|27|c6|e7|d6|ab|f0|2f|0f|78|a6|78|01|2c|ed|18|bb|47|34|54|ee|73|1b|99|83|ca|bf|d7|07|65|40|03|f2|75|98|d4|f3|22|6c|2f|77|1e|ec|e7|1e|bf|31|28|90|c4|0c|7a|37|cc|85|37|59|42|94|21|90|05|9e|e0|60|51|0a|83|b8|29|45|37|06|5a|5e|a3|b3|e0|08|cf|48|9b|51|17|7b|4c|fb|6d|06|e9|4f|dd|45|42|42|83|c2|f6|b3|70|29|86|08|c4|28|f7|b4|d6|1f|a1|63|94|f7|75|8f|ba|16|03|01|00|04|0e|00|00|00|
+---------+---------------+----------+
18:19:08,150,907 ETHER
|0 |1e|33|c8|aa|1b|40|8a|4b|47|20|ce|e4|08|00|45|00|05|99|39|1b|40|00|40|06|dc|9a|0a|c8|00|0d|0a|64|0a|71|15|b3|58|a7|62|61|d4|6e|00|36|c0|1a|50|18|72|10|25|35|00|00|16|03|01|00|51|02|00|00|4d|03|01|e9|4e|66|d1|56|43|c2|f8|df|95|62|d2|a1|8f|99|85|39|d3|22|4e|92|64|85|75|47|8f|29|f5|15|11|8e|10|20|b7|07|bc|84|e7|15|1e|6b|53|ed|6c|d5|1c|f6|15|10|ab|03|94|db|15|19|47|b7|43|04|30|59|fb|f2|52|e5|00|35|00|00|05|ff|01|00|01|00|16|03|01|05|0d|0b|00|05|09|00|05|06|00|05|03|30|82|04|ff|30|82|02|e7|a0|03|02|01|02|02|09|00|df|41|71|c6|48|ff|eb|19|30|0d|06|09|2a|86|48|86|f7|0d|01|01|0b|05|00|30|16|31|14|30|12|06|03|55|04|03|0c|0b|65|78|61|6d|70|6c|65|2e|63|6f|6d|30|1e|17|0d|31|38|30|32|30|34|31|38|31|33|35|36|5a|17|0d|32|38|30|32|30|32|31|38|31|33|35|36|5a|30|16|31|14|30|12|06|03|55|04|03|0c|0b|65|78|61|6d|70|6c|65|2e|63|6f|6d|30|82|02|22|30|0d|06|09|2a|86|48|86|f7|0d|01|01|01|05|00|03|82|02|0f|00|30|82|02|0a|02|82|02|01|00|e0|22|b0|41|79|e5|60|92|2c|c7|1c|97|84|f8|16|46|39|27|67|26|ad|d7|c8|dc|a4|18|2e|5a|1a|aa|f7|92|b0|71|8f|f8|a5|e0|d9|52|d3|6a|c9|e1|ae|7c|ce|f1|1c|92|99|dd|77|b7|f7|db|b3|2b|f4|96|23|d4|d8|08|e8|e2|f2|15|7b|41|16|30|3d|21|cd|5f|b0|3d|5a|0c|ec|60|9b|d8|78|99|32|c4|9f|3c|1b|50|d9|d4|cf|f0|4b|58|ca|c6|6f|56|f5|57|04|c9|d5|60|56|d2|25|66|ee|fe|f0|da|79|d6|f2|a5|67|08|7a|19|e6|82|90|5c|e0|a1|63|cf|4b|a8|43|c6|d3|08|a0|22|4a|df|be|42|22|8b|c9|6f|81|d1|b0|d8|e0|24|79|b3|3a|ad|10|af|bd|7a|43|56|a6|87|ad|d2|d4|96|cf|12|f9|9a|42|7a|c7|63|07|f7|cc|a9|a5|50|1e|bf|4d|2d|ae|e3|74|b7|a7|66|ca|ab|44|66|db|f5|e2|28|dd|a8|6f|76|0a|5a|d4|8d|42|c6|a0|16|c2|bd|d9|25|38|74|dc|bc|d3|95|03|b9|13|39|c0|4a|cd|d9|c9|cb|34|bf|9e|48|1e|e4|56|8f|47|9c|33|ed|eb|d6|db|5e|b7|46|85|d4|49|48|f3|24|ec|01|c3|98|74|50|f9|05|71|01|b5|96|f4|b5|bc|a3|57|78|3f|c7|4f|10|1e|da|96|59|c0|6d|c1|13|64|52|7b|f8|d7|88|59|29|ba|b7|5a|ec|a2|a7|1e|86|bb|ea|8b|f8|03|e3|73|b3|6e|70|27|29|b6|92|31|4f|52|73|f0|62|91|3e|90|53|40|d0|00|e2|1d|80|5e|82|a9|5e|fc|56|2f|76|d9|40|39|85|e3|4e|55|d2|a7|47|06|27|92|d9|ec|dd|7e|7d|c2|07|6d|73|9b|58|75|da|ba|cd|65|e3|d9|3b|da|78|4e|96|81|c1|d1|2f|f2|c0|65|e0|a2|12|33|1c|eb|0c|2a|10|c6|ea|87|4c|7a|a5|d7|c3|4c|a8|45|06|8d|ca|a5|90|7d|e0|57|be|e5|17|f0|13|41|28|6e|f6|f3|cb|b3|d4|9b|fa|b1|f9|30|a1|b4|2e|a4|de|73|39|35|82|32|de|55|f9|70|ee|ca|8a|d2|62|64|fc|fb|d7|56|17|c7|9a|3c|4f|3d|b1|4a|1d|37|fb|ad|40|ef|c4|14|cf|08|57|8e|68|5f|2b|d4|3b|cc|bb|23|9f|83|64|e3|22|39|53|30|5c|4d|0f|d8|4d|2a|5a|23|e6|ee|06|b6|38|5d|83|ef|91|f8|25|a6|e3|02|03|01|00|01|a3|50|30|4e|30|1d|06|03|55|1d|0e|04|16|04|14|bf|22|b8|2a|ec|f6|61|a9|05|50|d8|6a|f5|35|bd|32|3e|7c|9b|d7|30|1f|06|03|55|1d|23|04|18|30|16|80|14|bf|22|b8|2a|ec|f6|61|a9|05|50|d8|6a|f5|35|bd|32|3e|7c|9b|d7|30|0c|06|03|55|1d|13|04|05|30|03|01|01|ff|30|0d|06|09|2a|86|48|86|f7|0d|01|01|0b|05|00|03|82|02|01|00|35|e5|1e|a9|74|4f|78|59|c0|fd|da|a4|71|7a|4e|9d|b7|13|42|5b|d5|f3|95|1b|48|a6|91|c8|84|be|fc|67|ed|92|4e|2e|f2|e5|6e|14|38|a5|af|51|7f|54|11|5f|ed|15|bd|a6|d9|ef|9c|ef|ee|db|ed|fb|39|45|e2|5e|a9|53|66|73|40|98|27|38|4d|a3|1d|5f|eb|d3|f8|33|01|3d|ce|08|61|60|e1|ee|91|28|98|0a|a2|17|28|17|39|c1|af|1e|d2|f7|42|c6|5f|ff|a2|fa|77|2a|f0|4b|ed|4f|ea|d4|55|b4|4d|02|63|4f|9f|12|ab|86|26|a6|24|26|7e|26|e2|55|89|6b|7d|52|3c|48|ad|4c|0c|76|64|7d|a7|8f|ff|9e|23|28|4c|d5|79|85|ea|0c|ee|54|31|24|e2|66|53|4c|ae|a6|a6|43|c5|e1|16|c7|48|b3|af|49|c9|f6|30|e6|4a|da|2f|9b|1a|9d|5f|15|ff|ac|9e|c7|d3|e4|85|55|1d|d6|42|af|58|1e|57|d1|84|4f|87|a7|43|10|e8|13|42|fd|49|fa|c1|a2|ba|6a|30|a4|1e|fb|db|be|99|16|a1|dc|1e|45|45|54|c3|89|b8|b6|b8|74|90|2f|8b|8c|68|49|07|a2|7c|0d|a6|d6|3c|80|e2|94|66|b0|83|1f|e3|29|02|4d|76|44|1e|65|57|7c|9e|f2|01|0a|77|cd|c5|85|43|53|9e|e7|56|21|4b|55|05|78|0e|6b|7a|14|44|69|d8|cb|06|de|65|9a|0a|8b|15|11|4e|21|c6|1d|be|d4|6c|45|96|88|f2|04|d5|ff|03|fd|d3|24|7d|25|22|8b|22|37|99|4b|22|fa|8f|99|ec|71|88|58|8e|03|9f|76|19|a2|05|a6|3b|bc|87|d8|90|db|c1|cc|98|f9|1f|c5|00|96|e4|90|3d|ad|d0|fa|a1|d5|b6|71|74|7d|0c|a3|f2|79|5a|e6|31|fb|de|65|41|46|39|78|bd|d6|c1|b0|1a|44|16|a1|ed|27|c6|e7|d6|ab|f0|2f|0f|78|a6|78|01|2c|ed|18|bb|47|34|54|ee|73|1b|99|83|ca|bf|d7|07|65|40|03|f2|75|98|d4|f3|22|6c|2f|77|1e|ec|e7|1e|bf|31|28|90|c4|0c|7a|37|cc|85|37|59|42|94|21|90|05|9e|e0|60|51|0a|83|b8|29|45|37|06|5a|5e|a3|b3|e0|08|cf|48|9b|51|17|7b|4c|fb|6d|06|e9|4f|dd|45|42|42|83|c2|f6|b3|70|29|86|08|c4|28|f7|b4|d6|1f|a1|63|94|f7|75|8f|ba|16|03|01|00|04|0e|00|00|00|
+---------+---------------+----------+
18:19:08,556,603 ETHER
|0 |8a|4b|47|20|ce|e4|1e|33|c8|aa|1b|40|08|00|45|00|00|7c|05|fd|00|00|7f|06|15|d6|0a|64|0a|71|0a|c8|00|0d|58|a7|15|b3|00|36|bf|c6|62|61|d4|6e|50|18|16|d0|67|d0|00|00|16|03|01|00|4f|01|00|00|4b|03|03|6e|30|37|19|a3|c0|a4|d5|d1|3c|8b|ed|76|72|e2|9f|d5|5e|57|3e|9e|98|b7|33|a9|12|54|de|40|32|df|34|00|00|12|00|3d|00|35|00|3c|00|2f|00|b7|00|95|00|b6|00|94|00|ff|01|00|00|10|00|0d|00|0c|00|0a|06|01|05|01|04|01|03|01|02|01|
+---------+---------------+----------+
18:19:08,556,617 ETHER
|0 |1e|33|c8|aa|1b|40|8a|4b|47|20|ce|e4|08|00|45|00|00|28|39|1c|40|00|40|06|e2|0a|0a|c8|00|0d|0a|64|0a|71|15|b3|58|a7|62|61|d9|df|00|36|c0|1a|50|10|72|10|1f|c4|00|00|
+---------+---------------+----------+
18:19:08,683,863 ETHER
|0 |8a|4b|47|20|ce|e4|1e|33|c8|aa|1b|40|08|00|45|00|00|28|05|fe|00|00|7f|06|16|29|0a|64|0a|71|0a|c8|00|0d|58|a7|15|b3|00|36|c0|1a|62|61|d9|df|50|11|11|5f|13|df|00|00|
+---------+---------------+----------+
18:19:08,683,932 ETHER
|0 |1e|33|c8|aa|1b|40|8a|4b|47|20|ce|e4|08|00|45|00|00|28|39|1d|40|00|40|06|e2|09|0a|c8|00|0d|0a|64|0a|71|15|b3|58|a7|62|61|d9|df|00|36|c0|1b|50|11|72|10|1f|c4|00|00|
+---------+---------------+----------+
18:19:08,762,143 ETHER
|0 |8a|4b|47|20|ce|e4|1e|33|c8|aa|1b|40|08|00|45|00|00|28|05|ff|00|00|7f|06|16|28|0a|64|0a|71|0a|c8|00|0d|58|a7|15|b3|00|36|c0|1b|62|61|d9|e0|50|10|11|5e|13|df|00|00|
+---------+---------------+----------+
18:19:16,784,169 ETHER
|0 |8a|4b|47|20|ce|e4|1e|33|c8|aa|1b|40|08|00|45|00|00|2c|06|00|00|00|7f|06|16|23|0a|64|0a|71|0a|c8|00|0d|18|30|15|b3|00|36|e7|9f|00|00|00|00|60|02|16|d0|4b|f4|00|00|02|04|05|b4|
+---------+---------------+----------+
18:19:16,784,184 ETHER
|0 |1e|33|c8|aa|1b|40|8a|4b|47|20|ce|e4|08|00|45|00|00|2c|00|00|40|00|40|06|1b|23|0a|c8|00|0d|0a|64|0a|71|15|b3|18|30|98|bc|9a|61|00|36|e7|a0|60|12|72|10|1f|c8|00|00|02|04|05|b4|
+---------+---------------+----------+
18:19:16,886,424 ETHER
|0 |8a|4b|47|20|ce|e4|1e|33|c8|aa|1b|40|08|00|45|00|00|7c|06|01|00|00|7f|06|15|d2|0a|64|0a|71|0a|c8|00|0d|18|30|15|b3|00|36|e7|a0|98|bc|9a|62|50|18|16|d0|70|29|00|00|16|03|01|00|4f|01|00|00|4b|03|03|e5|7a|5a|74|be|6f|f0|54|cf|60|fd|45|d1|15|6e|25|28|82|23|11|ce|26|76|8f|3a|9b|db|bf|31|93|66|24|00|00|12|00|3d|00|35|00|3c|00|2f|00|b7|00|95|00|b6|00|94|00|ff|01|00|00|10|00|0d|00|0c|00|0a|06|01|05|01|04|01|03|01|02|01|
+---------+---------------+----------+
18:19:16,886,440 ETHER
|0 |1e|33|c8|aa|1b|40|8a|4b|47|20|ce|e4|08|00|45|00|00|28|6c|15|40|00|40|06|af|11|0a|c8|00|0d|0a|64|0a|71|15|b3|18|30|98|bc|9a|62|00|36|e7|f4|50|10|72|10|1f|c4|00|00|
+---------+---------------+----------+
18:19:16,886,658 ETHER
|0 |1e|33|c8|aa|1b|40|8a|4b|47|20|ce|e4|08|00|45|00|05|99|6c|16|40|00|40|06|a9|9f|0a|c8|00|0d|0a|64|0a|71|15|b3|18|30|98|bc|9a|62|00|36|e7|f4|50|18|72|10|25|35|00|00|16|03|01|00|51|02|00|00|4d|03|01|24|c5|4f|88|a2|2e|d6|95|40|f4|42|c1|97|78|b8|4a|33|43|32|94|d8|81|88|7f|09|44|df|3a|6c|79|d2|3a|20|1d|a7|32|d9|88|67|21|c6|69|f6|70|a7|20|28|0c|ac|a4|da|d0|95|3f|02|4f|51|68|78|7e|bf|1c|53|46|69|00|35|00|00|05|ff|01|00|01|00|16|03|01|05|0d|0b|00|05|09|00|05|06|00|05|03|30|82|04|ff|30|82|02|e7|a0|03|02|01|02|02|09|00|df|41|71|c6|48|ff|eb|19|30|0d|06|09|2a|86|48|86|f7|0d|01|01|0b|05|00|30|16|31|14|30|12|06|03|55|04|03|0c|0b|65|78|61|6d|70|6c|65|2e|63|6f|6d|30|1e|17|0d|31|38|30|32|30|34|31|38|31|33|35|36|5a|17|0d|32|38|30|32|30|32|31|38|31|33|35|36|5a|30|16|31|14|30|12|06|03|55|04|03|0c|0b|65|78|61|6d|70|6c|65|2e|63|6f|6d|30|82|02|22|30|0d|06|09|2a|86|48|86|f7|0d|01|01|01|05|00|03|82|02|0f|00|30|82|02|0a|02|82|02|01|00|e0|22|b0|41|79|e5|60|92|2c|c7|1c|97|84|f8|16|46|39|27|67|26|ad|d7|c8|dc|a4|18|2e|5a|1a|aa|f7|92|b0|71|8f|f8|a5|e0|d9|52|d3|6a|c9|e1|ae|7c|ce|f1|1c|92|99|dd|77|b7|f7|db|b3|2b|f4|96|23|d4|d8|08|e8|e2|f2|15|7b|41|16|30|3d|21|cd|5f|b0|3d|5a|0c|ec|60|9b|d8|78|99|32|c4|9f|3c|1b|50|d9|d4|cf|f0|4b|58|ca|c6|6f|56|f5|57|04|c9|d5|60|56|d2|25|66|ee|fe|f0|da|79|d6|f2|a5|67|08|7a|19|e6|82|90|5c|e0|a1|63|cf|4b|a8|43|c6|d3|08|a0|22|4a|df|be|42|22|8b|c9|6f|81|d1|b0|d8|e0|24|79|b3|3a|ad|10|af|bd|7a|43|56|a6|87|ad|d2|d4|96|cf|12|f9|9a|42|7a|c7|63|07|f7|cc|a9|a5|50|1e|bf|4d|2d|ae|e3|74|b7|a7|66|ca|ab|44|66|db|f5|e2|28|dd|a8|6f|76|0a|5a|d4|8d|42|c6|a0|16|c2|bd|d9|25|38|74|dc|bc|d3|95|03|b9|13|39|c0|4a|cd|d9|c9|cb|34|bf|9e|48|1e|e4|56|8f|47|9c|33|ed|eb|d6|db|5e|b7|46|85|d4|49|48|f3|24|ec|01|c3|98|74|50|f9|05|71|01|b5|96|f4|b5|bc|a3|57|78|3f|c7|4f|10|1e|da|96|59|c0|6d|c1|13|64|52|7b|f8|d7|88|59|29|ba|b7|5a|ec|a2|a7|1e|86|bb|ea|8b|f8|03|e3|73|b3|6e|70|27|29|b6|92|31|4f|52|73|f0|62|91|3e|90|53|40|d0|00|e2|1d|80|5e|82|a9|5e|fc|56|2f|76|d9|40|39|85|e3|4e|55|d2|a7|47|06|27|92|d9|ec|dd|7e|7d|c2|07|6d|73|9b|58|75|da|ba|cd|65|e3|d9|3b|da|78|4e|96|81|c1|d1|2f|f2|c0|65|e0|a2|12|33|1c|eb|0c|2a|10|c6|ea|87|4c|7a|a5|d7|c3|4c|a8|45|06|8d|ca|a5|90|7d|e0|57|be|e5|17|f0|13|41|28|6e|f6|f3|cb|b3|d4|9b|fa|b1|f9|30|a1|b4|2e|a4|de|73|39|35|82|32|de|55|f9|70|ee|ca|8a|d2|62|64|fc|fb|d7|56|17|c7|9a|3c|4f|3d|b1|4a|1d|37|fb|ad|40|ef|c4|14|cf|08|57|8e|68|5f|2b|d4|3b|cc|bb|23|9f|83|64|e3|22|39|53|30|5c|4d|0f|d8|4d|2a|5a|23|e6|ee|06|b6|38|5d|83|ef|91|f8|25|a6|e3|02|03|01|00|01|a3|50|30|4e|30|1d|06|03|55|1d|0e|04|16|04|14|bf|22|b8|2a|ec|f6|61|a9|05|50|d8|6a|f5|35|bd|32|3e|7c|9b|d7|30|1f|06|03|55|1d|23|04|18|30|16|80|14|bf|22|b8|2a|ec|f6|61|a9|05|50|d8|6a|f5|35|bd|32|3e|7c|9b|d7|30|0c|06|03|55|1d|13|04|05|30|03|01|01|ff|30|0d|06|09|2a|86|48|86|f7|0d|01|01|0b|05|00|03|82|02|01|00|35|e5|1e|a9|74|4f|78|59|c0|fd|da|a4|71|7a|4e|9d|b7|13|42|5b|d5|f3|95|1b|48|a6|91|c8|84|be|fc|67|ed|92|4e|2e|f2|e5|6e|14|38|a5|af|51|7f|54|11|5f|ed|15|bd|a6|d9|ef|9c|ef|ee|db|ed|fb|39|45|e2|5e|a9|53|66|73|40|98|27|38|4d|a3|1d|5f|eb|d3|f8|33|01|3d|ce|08|61|60|e1|ee|91|28|98|0a|a2|17|28|17|39|c1|af|1e|d2|f7|42|c6|5f|ff|a2|fa|77|2a|f0|4b|ed|4f|ea|d4|55|b4|4d|02|63|4f|9f|12|ab|86|26|a6|24|26|7e|26|e2|55|89|6b|7d|52|3c|48|ad|4c|0c|76|64|7d|a7|8f|ff|9e|23|28|4c|d5|79|85|ea|0c|ee|54|31|24|e2|66|53|4c|ae|a6|a6|43|c5|e1|16|c7|48|b3|af|49|c9|f6|30|e6|4a|da|2f|9b|1a|9d|5f|15|ff|ac|9e|c7|d3|e4|85|55|1d|d6|42|af|58|1e|57|d1|84|4f|87|a7|43|10|e8|13|42|fd|49|fa|c1|a2|ba|6a|30|a4|1e|fb|db|be|99|16|a1|dc|1e|45|45|54|c3|89|b8|b6|b8|74|90|2f|8b|8c|68|49|07|a2|7c|0d|a6|d6|3c|80|e2|94|66|b0|83|1f|e3|29|02|4d|76|44|1e|65|57|7c|9e|f2|01|0a|77|cd|c5|85|43|53|9e|e7|56|21|4b|55|05|78|0e|6b|7a|14|44|69|d8|cb|06|de|65|9a|0a|8b|15|11|4e|21|c6|1d|be|d4|6c|45|96|88|f2|04|d5|ff|03|fd|d3|24|7d|25|22|8b|22|37|99|4b|22|fa|8f|99|ec|71|88|58|8e|03|9f|76|19|a2|05|a6|3b|bc|87|d8|90|db|c1|cc|98|f9|1f|c5|00|96|e4|90|3d|ad|d0|fa|a1|d5|b6|71|74|7d|0c|a3|f2|79|5a|e6|31|fb|de|65|41|46|39|78|bd|d6|c1|b0|1a|44|16|a1|ed|27|c6|e7|d6|ab|f0|2f|0f|78|a6|78|01|2c|ed|18|bb|47|34|54|ee|73|1b|99|83|ca|bf|d7|07|65|40|03|f2|75|98|d4|f3|22|6c|2f|77|1e|ec|e7|1e|bf|31|28|90|c4|0c|7a|37|cc|85|37|59|42|94|21|90|05|9e|e0|60|51|0a|83|b8|29|45|37|06|5a|5e|a3|b3|e0|08|cf|48|9b|51|17|7b|4c|fb|6d|06|e9|4f|dd|45|42|42|83|c2|f6|b3|70|29|86|08|c4|28|f7|b4|d6|1f|a1|63|94|f7|75|8f|ba|16|03|01|00|04|0e|00|00|00|
+---------+---------------+----------+
18:19:16,933,784 ETHER
|0 |8a|4b|47|20|ce|e4|1e|33|c8|aa|1b|40|08|00|45|00|00|28|06|02|00|00|7f|06|16|25|0a|64|0a|71|0a|c8|00|0d|18|30|15|b3|00|36|e7|f4|98|bc|9f|d3|50|11|11|5f|30|2d|00|00|
+---------+---------------+----------+
18:19:16,933,874 ETHER
|0 |1e|33|c8|aa|1b|40|8a|4b|47|20|ce|e4|08|00|45|00|00|28|6c|17|40|00|40|06|af|0f|0a|c8|00|0d|0a|64|0a|71|15|b3|18|30|98|bc|9f|d3|00|36|e7|f5|50|11|72|10|1f|c4|00|00|
+---------+---------------+----------+
18:19:16,973,340 ETHER
|0 |8a|4b|47|20|ce|e4|1e|33|c8|aa|1b|40|08|00|45|00|00|28|06|03|00|00|7f|06|16|24|0a|64|0a|71|0a|c8|00|0d|18|30|15|b3|00|36|e7|f5|98|bc|9f|d4|50|10|11|5e|30|2d|00|00|
What does this error actually mean?
At the risk of stating the obvious it literally means "the handshake failed" and doesn't give you any more information than that. It could be caused by any number of possible issues. From your debug output we can see that:
The client issues a ClientHello to the server
The server responds with a ServerHello, a Certificate and a ServerHelloDone.
The server then attempts to read a response from the client but this read fails (most probably because the client has aborted - but without sending any kind of alert to indicate the cause of the problem).
Clearly something in the response that your server has sent back to the client has caused the client to just give up. The most likely culprit is the certificate, but its also possible that there is something about the ServerHello it doesn't like (e.g. the server didn't understand some extension that the client sent in its ClientHello, which it refuses to work without...although this is quite unlikely)
When looking in details to your debug log, it looks like a normal client/server handshake, in TLS 1.2. The last 9 bytes written mean the SERVER HELLO DONE is accomplished, and then the client (the last "read from") is sending 5 bytes that unfortunately we don't have. 5 bytes is a normal thing at this moment, these bytes are supposed to continue with the TLS exchange (Client key exchange,..) and they contain the "hello" byte, the protocol version (2 bytes), and the data length (2 bytes).
All this work would be easier for me if you provided a network capture. But OK, we have some information.
There is one reason to make openssl bug at this moment, when we receive 5 bytes instead of 11. This code will later generate the same handshake failure that you get :
* Request this many bytes in initial read.
* We can detect SSL 3.0/TLS 1.0 Client Hellos
* ('type == 3') correctly only when the following
* is in a single record, which is not guaranteed by
* the protocol specification:
* Byte Content
* 0 type \
* 1/2 version > record header
* 3/4 length /
* 5 msg_type \
* 6-8 length > Client Hello message
* 9/10 client_version /
*/
char buf_space[11];
But without any capture, it's almost impossible to say whether the client is sending a wrong Client key exchange, or if it's your openssl version which is faulty (more likely, it's the client).
Can you check what openssl version you are running ?
This appears to be a network or stack problem.
Assuming we are the server at 10.200.0.13 and the client is 10.100.10.113 your network dump decodes as follows:
18:19:07,529,418 receive connect request (SYN) from port 22695 to port 5555 with starting seq 0x0036bfc5
18:19:07,529,450 send accept (SYN ACK) with starting seq 0x6261d46d
18:19:07,668,343 receive segment containing ClientHello message with seq 0x0036bfc6 len 0x54
18:19:07,668,359 send ack (through 0x0036c01a)
18:19:07,668,617 send segment containing ServerHello, Certificate, ServerHelloDone with seq 0x6261d46e len 0x56+0x512+9=0x571
18:19:08,150,907 after no response ~0.5s, resend segment
18:19:08,556,603 receive resend of client 0x0036bfc6, which does NOT ack our data;
this implies client didn't get our data-ack and didn't get or accept (either copy of) our 0x6261d46e
18:19:08,556,617 send probe (ACK with no additional data)
18:19:08,683,863 receive probe from client, again neg-acking data but pos-acking SYN
18:19:08,683,932 after 2x neg-ack, send FIN (seq 0x6261d9df)
18:19:08,762,143 receive ACK of our FIN (and data?) (ack=0x6261d9e0)
18:19:16,784,169 receive new connect (SYN) from port 6192
...
This is screwy. If the client is any normal computer with an OS, usually those ack data at TCP level in the protocol stack even if their user program doesn't read (or accept) the data, at least up to some reasonable buffersize. The client is clearly getting at least some of our frames, since it correctly acks SYN and FIN, but it apparently refuses to ack our data.
If the client is something like a thermostat or an Arduino, where there is no actual OS but often instead a heap of code someone mashed together, often by copying from websites (including Stack?) without understanding it, so it doesn't work right. Or possibly the actual client is correct, but there is some broken middlebox like a firewall, IDS, 'smart' router, etc. in between.
Anyway if you can't get the TLS protocol messages, which are TCP data, delivered to the client at the TCP level, nothing else can work, so without any more useful info about or control over the client you're out of luck.
Sorry.
bad gethostbyaddr
A reverse DNS is not working well on your server. With your server IP, run these commands and check if they return anything (I suppose you're on Linux) :
dig -x XXX.XXX.XXX.XXX
host XXX.XXX.XXX.XXX
Check that your hosts file solves correctly your hostname. If you have to configure a DNS server (/etc/resolv.conf), do it.
We are having troubles with sending a 'large' SOAP request to one of our sources over SSL. When we send the same request, but with less data in it, it works without any problems. The small file is 10kb, the larger file is 30kb. The SOAP requests are send from OSB (11.1.1.4) in Weblogic (10.3.4).
Our source has checked what happens in their proxy and they see that the proxy waits for a part of the message, but never receives it:
[14/Sep/2016:11:04:56 +0200] [someurl.something.com/sid#7fcdebef0ae8][rid#7fcdebe080a0][/cxf/someServiceService-01-01][4] Input filter: Reading request body.
[14/Sep/2016:11:04:56 +0200] [someurl.something.com/sid#7fcdebef0ae8][rid#7fcdebe080a0][/cxf/someServiceService-01-01][9] Input filter: Bucket type HEAP contains 1 bytes.
[14/Sep/2016:11:04:56 +0200] [someurl.something.com/sid#7fcdebef0ae8][rid#7fcdebe080a0][/cxf/someServiceService-01-01][9] Input filter: Bucket type HEAP contains 8000 bytes.
[14/Sep/2016:11:04:56 +0200] [someurl.something.com/sid#7fcdebef0ae8][rid#7fcdebe080a0][/cxf/someServiceService-01-01][9] Input filter: Bucket type HEAP contains 192 bytes.
[14/Sep/2016:11:04:56 +0200] [someurl.something.com/sid#7fcdebef0ae8][rid#7fcdebe080a0][/cxf/someServiceService-01-01][9] Input filter: Bucket type HEAP contains 534 bytes.
[14/Sep/2016:11:04:56 +0200] [someurl.something.com/sid#7fcdebef0ae8][rid#7fcdebe080a0][/cxf/someServiceService-01-01][9] Input filter: Bucket type HEAP contains 7376 bytes.
[14/Sep/2016:11:05:38 +0200] [someurl.something.com/sid#7fcdebef0ae8][rid#7fcdebe080a0][/cxf/someServiceService-01-01][4] Error reading request body: The timeout specified has expired
We have HTTP dump on our server and from the logging I can see that the SSL handshaking is ok, we start sending our message, but it stops before it's completly done.
Padded plaintext before ENCRYPTION: len = 328
0000: 50 4F 53 54 20 2F 63 78 00 00 00 00 00 00 00 00 POST /cxf/
0010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 some
0020: 53 65 72 76 69 63 65 2D 30 31 2D 30 31 20 48 54 Service-01-01 HT
0030: 54 50 2F 31 2E 31 0D 0A 43 6F 6E 74 65 6E 74 2D TP/1.1..Content-
0040: 54 79 70 65 3A 20 74 65 78 74 2F 78 6D 6C 3B 20 Type: text/xml;
0050: 63 68 61 72 73 65 74 3D 75 74 66 2D 38 0D 0A 53 charset=utf-8..S
0060: 4F 41 50 41 63 74 69 6F 6E 3A 20 22 42 65 77 61 OAPAction: "
0070: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 someAction"..
0080: 55 73 65 72 2D 41 67 65 6E 74 3A 20 4A 61 76 61 User-Agent: Java
0090: 31 2E 36 2E 30 5F 33 31 0D 0A 48 6F 73 74 3A 20 1.6.0_31..Host:
00A0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 some.host
00B0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00C0: 6E 65 74 2E 62 65 0D 0A 41 63 63 65 70 74 3A 20 ..Accept:
00D0: 74 65 78 74 2F 68 74 6D 6C 2C 20 69 6D 61 67 65 text/html, image
00E0: 2F 67 69 66 2C 20 69 6D 61 67 65 2F 6A 70 65 67 /gif, image/jpeg
00F0: 2C 20 2A 2F 2A 3B 20 71 3D 2E 32 0D 0A 43 6F 6E , */*; q=.2..Con
0100: 6E 65 63 74 69 6F 6E 3A 20 4B 65 65 70 2D 41 6C nection: Keep-Al
0110: 69 76 65 0D 0A 43 6F 6E 74 65 6E 74 2D 4C 65 6E ive..Content-Len
0120: 67 74 68 3A 20 34 32 35 39 36 0D 0A 0D 0A CB F2 gth: 42596......
0130: DB 39 D1 16 D4 4C D3 05 BB 08 3C 2B A0 1E 39 BF .9...L....<+..9.
0140: A9 15 05 05 05 05 05 05 ........
Padded plaintext before ENCRYPTION: len = 16128
0000: 3F 78 6D 6C 20 76 65 72 73 69 6F 6E 3D 22 31 2E ?xml version="1.
0010: 30 22 20 65 6E 63 6F 64 69 6E 67 3D 22 55 54 46 0" encoding="UTF
0020: 2D 38 22 3F 3E 0A 3C 73 6F 61 70 65 6E 76 3A 45 -8"?>.<soapenv:E
0030: 6E 76 65 6C 6F 70 65 20 78 6D 6C 6E 73 3A 73 6F nvelope xmlns:so
0040: 61 70 65 6E 76 3D 22 68 74 74 70 3A 2F 2F 73 63 apenv="http://sc
... and so forth
This ends before I've seen the complete SOAP request pass through. And we receive:
Padded plaintext after DECRYPTION: len = 328
0000: 48 54 54 50 2F 31 2E 31 20 34 30 30 20 42 61 64 HTTP/1.1 400 Bad
0010: 20 52 65 71 75 65 73 74 0D 0A 44 61 74 65 3A 20 Request..Date:
0020: 4D 6F 6E 2C 20 31 30 20 4F 63 74 20 32 30 31 36 Mon, 10 Oct 2016
0030: 20 30 37 3A 35 32 3A 30 37 20 47 4D 54 0D 0A 53 07:52:07 GMT..S
0040: 65 72 76 65 72 3A 20 41 70 61 63 68 65 0D 0A 53 erver: Apache..S
0050: 74 72 69 63 74 2D 54 72 61 6E 73 70 6F 72 74 2D trict-Transport-
0060: 53 65 63 75 72 69 74 79 3A 20 6D 61 78 2D 61 67 Security: max-ag
0070: 65 3D 33 31 35 33 36 30 30 30 3B 20 69 6E 63 6C e=31536000; incl
0080: 75 64 65 53 75 62 44 6F 6D 61 69 6E 73 0D 0A 4C udeSubDomains..L
0090: 61 73 74 2D 4D 6F 64 69 66 69 65 64 3A 20 54 75 ast-Modified: Tu
00A0: 65 2C 20 30 33 20 4D 61 72 20 32 30 31 35 20 31 e, 03 Mar 2015 1
00B0: 32 3A 32 35 3A 32 31 20 47 4D 54 0D 0A 45 54 61 2:25:21 GMT..ETa
00C0: 67 3A 20 22 62 65 38 2D 35 31 30 36 31 36 64 61 g: "be8-510616da
00D0: 38 63 62 31 64 22 0D 0A 41 63 63 65 70 74 2D 52 8cb1d"..Accept-R
00E0: 61 6E 67 65 73 3A 20 62 79 74 65 73 0D 0A 43 6F anges: bytes..Co
00F0: 6E 74 65 6E 74 2D 4C 65 6E 67 74 68 3A 20 33 30 ntent-Length: 30
0100: 34 38 0D 0A 43 6F 6E 6E 65 63 74 69 6F 6E 3A 20 48..Connection:
0110: 63 6C 6F 73 65 0D 0A 43 6F 6E 74 65 6E 74 2D 54 close..Content-T
0120: 79 70 65 3A 20 74 65 78 74 2F 68 74 6D 6C 0D 0A ype: text/html..
0130: 0D 0A 7E 01 14 86 D8 1F DA 05 97 49 26 2B 2F 65 ...........I&+/e
0140: DB 5E ED 05 F2 AA 01 01 .^......
Our server team has already tried increasing the timeouts in weblogic under Servers - Protocols - HTTP by a factor of 10, but with no success.
Increasing any possible timeout settings in OSB didn't help.
We are pretty sure it is a Weblogic (maybe OSB) issue as sending the larger request from the same server with curl doesn't give any problems.
Our Development environment does NOT have this issue. The problem is, we currently have an upgraded Weblogic (10.3.6)/OSB (11.1.1.7) installed there for an upcoming upgrade of the software on other environments. Same configuration though.
Any insight on what could be going wrong or what we could try would be helpfull. Let me know if you need any additional information.
I'm making an HTTP POST request using curl like this:
curl -v -i -d 'var=something' 'http://mysite.com/whatever.json'
How can I see the body of the outgoing request?
Motivation: I'm debugging an Objective-C HTTP library. POST requests that are failing in this library are working from the Terminal using curl. I think comparing the bodies of the requests will help debug.
You can use --trace or --trace-ascii to get an traffic dump:
curl --trace - -d 'var=something' 'http://mysite.com/whatever.json'
[...]
=> Send header, 246 bytes (0xf6)
0000: 50 4f 53 54 20 2f 77 68 61 74 65 76 65 72 2e 6a POST /whatever.j
0010: 73 6f 6e 20 48 54 54 50 2f 31 2e 31 0d 0a 55 73 son HTTP/1.1..Us
0020: 65 72 2d 41 67 65 6e 74 3a 20 63 75 72 6c 2f 37 er-Agent: curl/7
0030: 2e 32 31 2e 36 20 28 78 38 36 5f 36 34 2d 70 63 .21.6 (x86_64-pc
0040: 2d 6c 69 6e 75 78 2d 67 6e 75 29 20 6c 69 62 63 -linux-gnu) libc
0050: 75 72 6c 2f 37 2e 32 31 2e 36 20 4f 70 65 6e 53 url/7.21.6 OpenS
0060: 53 4c 2f 31 2e 30 2e 30 65 20 7a 6c 69 62 2f 31 SL/1.0.0e zlib/1
0070: 2e 32 2e 33 2e 34 20 6c 69 62 69 64 6e 2f 31 2e .2.3.4 libidn/1.
0080: 32 32 20 6c 69 62 72 74 6d 70 2f 32 2e 33 0d 0a 22 librtmp/2.3..
0090: 48 6f 73 74 3a 20 6d 79 73 69 74 65 2e 63 6f 6d Host: mysite.com
00a0: 0d 0a 41 63 63 65 70 74 3a 20 2a 2f 2a 0d 0a 43 ..Accept: */*..C
00b0: 6f 6e 74 65 6e 74 2d 4c 65 6e 67 74 68 3a 20 31 ontent-Length: 1
00c0: 33 0d 0a 43 6f 6e 74 65 6e 74 2d 54 79 70 65 3a 3..Content-Type:
00d0: 20 61 70 70 6c 69 63 61 74 69 6f 6e 2f 78 2d 77 application/x-w
00e0: 77 77 2d 66 6f 72 6d 2d 75 72 6c 65 6e 63 6f 64 ww-form-urlencod
00f0: 65 64 0d 0a 0d 0a ed....
=> Send data, 13 bytes (0xd)
0000: 76 61 72 3d 73 6f 6d 65 74 68 69 6e 67 var=something
[...]
--trace-ascii has less detail:
curl --trace-ascii - -d 'var=something' 'http://mysite.com/whatever.json'
[...]
=> Send header, 246 bytes (0xf6)
0000: POST /whatever.json HTTP/1.1
001e: User-Agent: curl/7.21.6 (x86_64-pc-linux-gnu) libcurl/7.21.6 Ope
005e: nSSL/1.0.0e zlib/1.2.3.4 libidn/1.22 librtmp/2.3
0090: Host: mysite.com
00a2: Accept: */*
00af: Content-Length: 13
00c3: Content-Type: application/x-www-form-urlencoded
00f4:
=> Send data, 13 bytes (0xd)
0000: var=something
[...]
You won't get friendly parsing, but you could always use tcpdump or Wireshark to see exactly what's being sent back and forth.
If you have it available, setting up netcat to listen on a port and then pointing both curl and your failing requests at that host/port would also work.
http://www.faqs.org/rfcs/rfc4253.html
::http://www.faqs.org/rfcs/rfc4251.html
Does anyone know the SSH key file format?
!! UPDATE
!! The base64 function I was using doesn't work. After running the key file through a different function (mozilla's built in atob()) the data fit into the specifications listed above.
I have rsa key files created with puttygen, but I must be missing something critical. Here is the hex of the public section:
00 00 00 07 73 73 68 2d 72 73 61 00 00 00 01 25
00 00 00 10 1c 1c 57 3f 4f 58 63 69 38 ad 19 35
b5 28 3d 78 53 35 53 6c 15 0e 69 23 ac 17 14 84
21 29 13 07 36 62 90 26 37 93 73 17 28 b8 ce 95
c3 11 24 21 61 b7 82 d9 04 42 97 f8 27 c3 44 06
46 ca e8 a3 a3 34 d7 3c c3 95 13 dd 16 1b 2c 29
7c 35 19 5f c2 7a 17 d5 14 0d 26 36 27 18 71 67
8d 9c 5b c4 7d
first 4 bytes are UINT 7, the number of bytes inthe following string "ssh-rsa" but the format stops making sense after that. It SHOULD be followed by two MPINT but they lengths don't add up for the 3rd value.
Thanks!
It's in ASN.1 syntax. You should be able to use an ASN.1 parser along with the specs for the file contents to decipher it.