Through smart card using a tool, I am signing data "Hello".
The output what i am getting is in hex format.
14 5F 65 CE 7C 2D 8A 0A FA B0 FB 86 CE 28 90 84
37 2D 04 63 B2 35 FA 40 4A B6 35 C8 90 AF 55 7F
B1 CA FE FD 5B F9 1B 7C DB 74 63 BF 16 5B B3 6D
E8 2D B6 D7 2E 90 AF 0A 5E CF 78 73 E3 37 02 C2
97 0E F9 B3 40 4C 67 CD E4 7C D9 4B D3 C9 86 51
8E 1E 84 81 B4 30 AC 68 96 59 CB 63 E5 C8 28 48
C7 1D E8 E9 FC E8 C9 BE 36 33 0A F0 A9 35 C4 D4
BF 60 66 21 5C 41 8F 48 91 D4 BB AF 75 75 7A B3
2A 8A 28 B8 30 D1 B4 6B 69 23 82 2D 28 77 30 05
D5 C9 AB 41 17 C1 68 6D D9 80 0F F2 C1 FC 32 6E
22 61 27 97 9C DD C3 50 33 AA DB F4 BA 98 29 FA
4F E2 B4 BC C5 9E 90 34 F3 BC 3D 78 01 47 AF 96
20 06 6F F9 41 30 D7 35 52 D3 DE 85 E3 FE 0B B7
15 4D 1A 73 B8 36 F4 A1 59 A2 7E 05 50 8B 52 AC
B4 EF 2D D9 29 9D D9 BB C8 DF F3 67 C5 D1 D9 C0
0C 65 68 A8 12 9B 24 92 4E EB 98 D8 B0 D9 2E 6A
The respective signed data in string format I have saved in file signedData.txt.
._eÎ|-Š.ú°û†Î(„7-.c²5ú#J¶5ȯU±Êþý[ù.|Ûtc¿.[³mè-¶×.¯.^Ïxsã7.—.ù³#LgÍä|ÙKÓɆQŽ.„´0¬h– YËcåÈ(HÇ.èéüèɾ63.ð©5ÄÔ¿`f!\AH‘Ô»¯uuz³*Š(¸0Ñ´ki#‚-(w0.ÕÉ«A.ÁhmÙ€.òÁü2n"a'—œÝÃP3ªÛôº˜)úOâ ´¼Åž4ó¼=x.G¯– .oùA0×5RÓÞ…ãþ.·.M.s¸6ô¡Y¢~.P‹R¬´ï-Ù)Ù»ÈßógÅÑÙÀ.eh¨.›$’Në˜Ø°Ù.j
I am using Windows CryptoAPI for verification of this signed data.
I am opening the file signedData.txt and writing the data into the buffer "
signedBuffer"
Then i am finding my certificate from the store and taking the handle of public key using
hPubKey = CryptImportPublicKeyInfo(hProv, ENCODING_TYPE, &pCertContext->pCertInfo->SubjectPublicKeyInfo, &hCertPubKey);
Then I am creating a hash object using CALG_SHA_256
hObject = CryptCreateHash(hProv, CALG_SHA_256, 0, 0, &hHashObject);
Then adding my data with the hash object
CryptHashData(hhObject, Buffer, BufferLen, 0); // char Buffer = "Hello";
// int BufferLen = strlen(Buffer );
In the last step i am verifying the signature using
CryptVerifySignature(hObject,signedBuffer,signedBufferBytes,hPubKey,NULL,0); /* signedBuffer contains signed data in string format */
/*signedBufferBytes has no. of bytes in the buffer */
but this signed data is not verified. Error code i am getting is 0x80090006 - Invalid Signature.
Public key In hex format is :
30 82 01 0a 02 82 01 01 00 b8 f8 dc 2c a5 03 84
ba 72 c6 0e 03 89 51 6f 39 a8 41 e3 49 b3 f7 14
31 d3 43 b7 fc 1f 61 c2 43 b0 77 9e 19 af f4 8b
02 99 72 c1 17 21 1d 23 da ab 53 54 74 33 e4 ab
9d 82 d2 68 33 9a b5 9c 99 cb f0 12 e0 f8 44 4f
e8 91 3f 60 ed ca fa 3b 40 bd 64 50 92 d3 c2 c1
48 ad 24 3e ca 64 2c 50 a9 01 b5 9f f4 a4 46 e5
84 e9 a4 87 41 86 a1 7a 7f fc a6 f0 e0 b1 de f0
c1 f2 5d c8 84 16 15 4d e4 df 43 43 3a cd ad ec
eb af 1b 9c a7 5c 40 dc ae 1f 71 6e a4 c6 0f dd
3e 3c c8 0d 25 4c 61 74 df aa ed b5 d5 b9 06 6a
8e b0 b7 c0 e6 c9 bf db b1 07 2e a2 76 aa e7 28
1c 8d 32 4e b3 58 1d 34 89 96 ed 3e da 29 e0 1e
c9 c2 2e 18 19 a6 ba 91 32 b7 85 97 87 92 16 c5
01 b4 4f 57 5c 56 1b f5 f4 6a 29 6b 2e 51 8b f5
4c 6f b8 fd cb 09 d9 fd 66 09 04 49 b6 ba 7e d0
af 70 3a 51 41 5a a5 04 bf 02 03 01 00 01
The Signature buffer now I am using is:
BYTE bSignatureBuf[] = {
0x6A, 0x2E, 0xD9, 0xB0, 0xD8, 0x98, 0xEB, 0x4E, 0x92, 0x24, 0x9B, 0x12, 0xA8, 0x68, 0x65, 0x0C,
0xC0, 0xD9, 0xD1, 0xC5, 0x67, 0xF3, 0xDF, 0xC8, 0xBB, 0xD9, 0x9D, 0x29, 0xD9, 0x2D, 0xEF, 0xB4,
0xAC, 0x52, 0x8B, 0x50, 0x05, 0x7E, 0xA2, 0x59, 0xA1, 0xF4, 0x36, 0xB8, 0x73, 0x1A, 0x4D, 0x15,
0xB7, 0x0B, 0xFE, 0xE3, 0x85, 0xDE, 0xD3, 0x52, 0x35, 0xD7, 0x30, 0x41, 0xF9, 0x6F, 0x06, 0x20,
0x96, 0xAF, 0x47, 0x01, 0x78, 0x3D, 0xBC, 0xF3, 0x34, 0x90, 0x9E, 0xC5, 0xBC, 0xB4, 0xE2, 0x4F,
0xFA, 0x29, 0x98, 0xBA, 0xF4, 0xDB, 0xAA, 0x33, 0x50, 0xC3, 0xDD, 0x9C, 0x97, 0x27, 0x61, 0x22,
0x6E, 0x32, 0xFC, 0xC1, 0xF2, 0x0F, 0x80, 0xD9, 0x6D, 0x68, 0xC1, 0x17, 0x41, 0xAB, 0xC9, 0xD5,
0x05, 0x30, 0x77, 0x28, 0x2D, 0x82, 0x23, 0x69, 0x6B, 0xB4, 0xD1, 0x30, 0xB8, 0x28, 0x8A, 0x2A,
0xB3, 0x7A, 0x75, 0x75, 0xAF, 0xBB, 0xD4, 0x91, 0x48, 0x8F, 0x41, 0x5C, 0x21, 0x66, 0x60, 0xBF,
0xD4, 0xC4, 0x35, 0xA9, 0xF0, 0x0A, 0x33, 0x36, 0xBE, 0xC9, 0xE8, 0xFC, 0xE9, 0xE8, 0x1D, 0xC7,
0x48, 0x28, 0xC8, 0xE5, 0x63, 0xCB, 0x59, 0x96, 0x68, 0xAC, 0x30, 0xB4, 0x81, 0x84, 0x1E, 0x8E,
0x51, 0x86, 0xC9, 0xD3, 0x4B, 0xD9, 0x7C, 0xE4, 0xCD, 0x67, 0x4C, 0x40, 0xB3, 0xF9, 0x0E, 0x97,
0xC2, 0x02, 0x37, 0xE3, 0x73, 0x78, 0xCF, 0x5E, 0x0A, 0xAF, 0x90, 0x2E, 0xD7, 0xB6, 0x2D, 0xE8,
0x6D, 0xB3, 0x5B, 0x16, 0xBF, 0x63, 0x74, 0xDB, 0x7C, 0x1B, 0xF9, 0x5B, 0xFD, 0xFE, 0xCA, 0xB1,
0x7F, 0x55, 0xAF, 0x90, 0xC8, 0x35, 0xB6, 0x4A, 0x40, 0xFA, 0x35, 0xB2, 0x63, 0x04, 0x2D, 0x37,
0x84, 0x90, 0x28, 0xCE, 0x86, 0xFB, 0xB0, 0xFA, 0x0A, 0x8A, 0x2D, 0x7C, 0xCE, 0x65, 0x5F, 0x14
};
First of all, you should not treat your binary data as text, as given in the comments.
Even more important is that your signature has been generated using PKCS#1 SHA-1 and your hash function specifies SHA-256 - presumably using PKCS#1 [^1]. You have to use identical hash function during signature generation and verification.
You can see the hash function used by looking up the OID in the ASN.1 description after decrypting your signature with the public key (in case you are wondering how to find out the hash function used).
Also note this little remark from the API:
The native cryptography API uses little-endian byte order (ed: for the signature bytes)
note that the non-M$ world uses big-endian byte order.
[^1]: as usual Microsoft does not specify the exact protocols they are applying [^2]
[^2]: this should view correctly once stackoverflow makes the tiny effort to implement markdown footnotes
Related
Please, can someone tell me what I'm doing wrong. I have to AES authenticate my card. The card is Mifare DESFire EV1 4K and the reader is Omnikey 5121. I followed some examples here on Stack but I always fail in the last step where card's rotated RndA is not equal to my rotated RndA. Is something wrong with the AES configuration?
public static byte[] Authenticate_AES(this SCardReader reader, byte[] key)
{
using (var aes = Aes.Create())
{
aes.Key = key;
aes.Padding = PaddingMode.None;
aes.Mode = CipherMode.CBC;
aes.BlockSize = 128;
aes.IV = SCardUtils.StringToByteArray("00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00");
var decryptor = aes.CreateDecryptor();
var encryptor = aes.CreateEncryptor();
var rnd = new Random();
//Get encrypted RandB from the card
var rAPDU = reader.Transmit(0x90, 0xAA, 0, 0, SCardUtils.StringToByteArray("00"));
if (!rAPDU.HasData)
throw new Exception("RandB_enc is null");
//Encrypted RndB
var RndB_enc = new byte[16];
rAPDU.Data.CopyTo(RndB_enc, 0);
SCardUtils.ShowBytes(RndB_enc, "RndB_enc");
//Decrypt encrypted RndB
var RndB = decryptor.TransformFinalBlock(RndB_enc, 0, RndB_enc.Length);
SCardUtils.ShowBytes(RndB, "RndB");
//Rotate RndB 1 byte to the left
var RndB_rot = SCardUtils.RotateLeft(RndB);
SCardUtils.ShowBytes(RndB_rot, "RndB_rot");
//Generate random RndA
var RndA = new byte[16];
rnd.NextBytes(RndA);
SCardUtils.ShowBytes(RndA, "RndA");
//Concatenate RndA and RndB_rot
var RndAB_rot = RndA.Concat(RndB_rot).ToArray();
SCardUtils.ShowBytes(RndAB_rot, "RndAB_rot");
//Encypt RndAB_rot
var dataToSend = encryptor.TransformFinalBlock(RndAB_rot, 0, RndAB_rot.Length);
SCardUtils.ShowBytes(dataToSend, "Encrypted data");
rAPDU = reader.Transmit(0x90, 0xAF, 0, 0, dataToSend);
if (!rAPDU.HasData)
throw new Exception("rAPDU data is null");
//Encrypted RndA_rot
var RndA_rot_enc = new byte[16];
rAPDU.Data.CopyTo(RndA_rot_enc, 0);
SCardUtils.ShowBytes(RndA_rot_enc, "RndA_rot_enc");
//Decrypt encrypted RndA_rot
var RndA_rot_dec = decryptor.TransformFinalBlock(RndA_rot_enc, 0, RndA_rot_enc.Length);
SCardUtils.ShowBytes(RndA_rot_dec, "RndA_rot_dec");
var RndA_rot = SCardUtils.RotateLeft(RndA);
SCardUtils.ShowBytes(RndA_rot, "RndA_rot");
//Compare RndA_rot with RndA rotated to the left
if (!SCardUtils.IsEqualTo(RndA_rot_dec, RndA_rot))
throw new Exception($"Error authenticating card. The values are not equal.");
var sessionKey = new byte[16];
Array.Copy(RndA, 0, sessionKey, 0, 4);
Array.Copy(RndB, 0, sessionKey, 4, 4);
Array.Copy(RndA, 12, sessionKey, 8, 4);
Array.Copy(RndB, 12, sessionKey, 12, 4);
return sessionKey;
}
}
Here is the output:
Reader name: OMNIKEY CardMan 5x21-CL 0
RndB_enc: 08 DD A2 12 57 43 6C F7 75 98 78 9E 6C 0A A7 06
RndB: 16 C8 35 7A 4A 36 29 D2 F0 86 26 AD FA CA 81 9F
RndB_rot: C8 35 7A 4A 36 29 D2 F0 86 26 AD FA CA 81 9F 16
RndA: 77 93 A5 8D 0E 0D 88 88 22 C3 40 9C 26 67 95 35
RndAB_rot: 77 93 A5 8D 0E 0D 88 88 22 C3 40 9C 26 67 95 35 C8 35 7A 4A 36 29 D2 F0 86 26 AD FA CA 81 9F 16
Data: 4D BD 7A E8 B8 6C 00 5F E4 B5 B5 42 7B AE 51 39 25 77 CB 60 83 6A E8 15 B9 9D FD A9 FD A7 75 9F
RndA_rot_enc: D6 CB CF 08 5F 8A E8 6C 30 95 34 6F DD CF 4F FA
RndA_rot_dec: 6B 70 54 39 CD 8E 97 42 E2 A5 FF E3 90 95 46 E0
RndA_rot: 93 A5 8D 0E 0D 88 88 22 C3 40 9C 26 67 95 35 77
So you can see RndA_rot_dec and RndA_rot are not equal and I can't figure it out why.
Thank you all in advance.
I finally got the solution. I also modified the code a bit so it can be more understandable for everyone who will have a problem as I did. For several weeks.
public static byte[] Authenticate_AES(this SCardReader reader, byte[] key, byte[] IV)
{
//Get encrypted RndB from the tag
var rAPDU = reader.Transmit(0x90, 0xAA, 0, 0, SCardUtils.StringToByteArray("00"));
if (!rAPDU.HasData)
throw new Exception("RandB_enc is null");
var aes = Aes.Create();
aes.Mode = CipherMode.CBC;
aes.KeySize = 128;
aes.BlockSize = 128;
aes.Padding = PaddingMode.None;
aes.Key = key;
aes.IV = IV; //16 bytes of zeros
//Encrypted RndB from the tag
var RndB_enc = rAPDU.Data.ToArray();
SCardUtils.ShowBytes(RndB_enc, "RndB_enc");
var decryptor = aes.CreateDecryptor();
//Decrypt encrypted RndB
var RndB = decryptor.TransformFinalBlock(RndB_enc, 0, RndB_enc.Length);
SCardUtils.ShowBytes(RndB, "RndB");
//Rotate RndB 1 byte to the left
var RndB_rot = SCardUtils.RotateLeft(RndB);
SCardUtils.ShowBytes(RndB_rot, "RndB_rot");
var rnd = new Random();
//Generate random RndA
var RndA = new byte[16];
rnd.NextBytes(RndA);
SCardUtils.ShowBytes(RndA, "RndA");
//Concatenate RndA and RndB_rot
var RndAB_rot = RndA.Concat(RndB_rot).ToArray();
SCardUtils.ShowBytes(RndAB_rot, "RndAB_rot");
//IV is now encrypted RndB received from the tag
aes.IV = RndB_enc;
var encryptor = aes.CreateEncryptor();
//Encypt RndAB_rot
var RndAB_rot_enc = encryptor.TransformFinalBlock(RndAB_rot, 0, RndAB_rot.Length);
SCardUtils.ShowBytes(RndAB_rot_enc, "RndAB_rot_enc");
rAPDU = reader.Transmit(0x90, 0xAF, 0, 0, RndAB_rot_enc);
if (!rAPDU.HasData)
throw new Exception("rAPDU data is null");
//Encrypted RndA_rot from the tag
var RndA_rot_enc = rAPDU.Data.ToArray();
SCardUtils.ShowBytes(RndA_rot_enc, "RndA_rot_enc");
//IV is now the last 16 bytes of RndAB_rot_enc
aes.IV = RndAB_rot_enc.Skip(16).Take(16).ToArray();
decryptor = aes.CreateDecryptor();
//Decrypt encrypted RndA_rot
var RndA_rot = decryptor.TransformFinalBlock(rAPDU.Data, 0, rAPDU.Data.Length);
SCardUtils.ShowBytes(RndA_rot, "RndA_rot");
//Compare RndA_rot_dec with RndA_rot
if (!SCardUtils.IsEqualTo(RndA_rot, SCardUtils.RotateLeft(RndA)))
throw new Exception($"Error authenticating card. The values are not equal.");
var sessionKey = new byte[16];
Array.Copy(RndA, 0, sessionKey, 0, 4);
Array.Copy(RndB, 0, sessionKey, 4, 4);
Array.Copy(RndA, 12, sessionKey, 8, 4);
Array.Copy(RndB, 12, sessionKey, 12, 4);
aes.Clear();
return sessionKey;
}
The problem was with Init Vector. It is changing through the three-pass authentication.
Here is the final output:
Reader name: OMNIKEY CardMan 5x21-CL 0
RndB_enc: 6F 40 6D 9D 51 7A 2C 9E 88 C9 2C 84 80 94 E3 F7
RndB: FB FB A7 5F 54 97 D7 CA 4B 15 07 F1 A0 D1 A2 68
RndB_rot: FB A7 5F 54 97 D7 CA 4B 15 07 F1 A0 D1 A2 68 FB
RndA: 87 7B BA 48 4D 01 14 CE 4D 8E 33 A5 1B 0F 00 E9
RndAB_rot: 87 7B BA 48 4D 01 14 CE 4D 8E 33 A5 1B 0F 00 E9 FB A7 5F 54 97 D7 CA 4B 15 07 F1 A0 D1 A2 68 FB
RndAB_rot_enc: 0B 8B 4E D9 BF 40 51 F0 83 FC 44 E8 B7 A7 21 26 DE A9 B9 CE E5 05 F7 A5 46 FE 91 0F 59 2B 90 E7
RndA_rot_enc: 9F 3D E8 90 37 2E 7B F6 1E BA AC 29 6E 94 1C 9E
RndA_rot: 7B BA 48 4D 01 14 CE 4D 8E 33 A5 1B 0F 00 E9 87
As you can see RndA (line 5) rotated 1 byte to the left is equal to RndA_rot (last line). With this authentication is completed.
On https://tls.ulfheim.net/ there is an example showing how to compute a signature in the section of " Server Key Exchange ".
https://i.ibb.co/Y7fbkDw/1.jpg (This image shows the Server Key Exchange section on the website that I refer to. Could not embed the image in this post.)
Whatever I try I dont get the same output as the one on that website, I dont understand why.
I've tried storing the same data in two different ways, then use the same openssl command that they use on their example. None of the methods gave the same output.
Method 1.
char hex[] = { 0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08, 0x09, 0x0a, 0x0b, 0x0c, 0x0d, 0x0e, 0x0f, 0x10, 0x11, 0x12, 0x13, 0x14, 0x15, 0x16, 0x17, 0x18, 0x19, 0x1a, 0x1b, 0x1c, 0x1d, 0x1e, 0x1f, 0x70, 0x71, 0x72, 0x73, 0x74, 0x75, 0x76, 0x77, 0x78, 0x79, 0x7a, 0x7b, 0x7c, 0x7d, 0x7e, 0x7f, 0x80, 0x81, 0x82, 0x83, 0x84, 0x85, 0x86, 0x87, 0x88, 0x89, 0x8a, 0x8b, 0x8c, 0x8d, 0x8e, 0x8f, 0x20, 0x9f, 0xd7, 0xad, 0x6d, 0xcf, 0xf4, 0x29, 0x8d, 0xd3, 0xf9, 0x6d, 0x5b, 0x1b, 0x2a, 0xf9, 0x10, 0xa0, 0x53, 0x5b, 0x14, 0x88, 0xd7, 0xf8, 0xfa, 0xbb, 0x34, 0x9a, 0x98, 0x28, 0x80, 0xb6, 0x15 };
ofstream myfile("c:/hex1.txt", ios::binary);
myfile.write(hex, sizeof hex);
then:
openssl dgst -hex -sign server.key -sha256 hex1.txt
Method 2.
I had this data stored in hex2.txt (as ASCII):
\x00\x01\x02\x03\x04\x05\x06\x07\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f\x10\x11\x12\x13\x14\x15\x16\x17\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f\x70\x71\x72\x73\x74\x75\x76\x77\x78\x79\x7a\x7b\x7c\x7d\x7e\x7f\x80\x81\x82\x83\x84\x85\x86\x87\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f\x20\x9f\xd7\xad\x6d\xcf\xf4\x29\x8d\xd3\xf9\x6d\x5b\x1b\x2a\xf9\x10\xa0\x53\x5b\x14\x88\xd7\xf8\xfa\xbb\x34\x9a\x98\x28\x80\xb6\x15
then:
openssl dgst -hex -sign server.key -sha256 hex2.txt
Method 1
You left out curve_info. The applicable RFC 4492 section 5.4 updated for TLS1.2 by RFC 5246 appendix A.7 actually defines the signature to be over in effect client_random + server_random + SKX_params where SKX_params is type ServerECDHParams and consists of ECParameters curve_params and ECPoint public -- these are what ulfheim labels Curve Info and Public Key.
With the correct data I get the correct result:
$ od -tx1 70148855.bin
0000000 00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f
0000020 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f
0000040 70 71 72 73 74 75 76 77 78 79 7a 7b 7c 7d 7e 7f
0000060 80 81 82 83 84 85 86 87 88 89 8a 8b 8c 8d 8e 8f
0000100 03 00 1d 20 9f d7 ad 6d cf f4 29 8d d3 f9 6d 5b
0000120 1b 2a f9 10 a0 53 5b 14 88 d7 f8 fa bb 34 9a 98
0000140 28 80 b6 15
$ openssl sha256 <70148855.bin -sign $privkey -hex
(stdin)= 0402b661f7c191ee59be45376639bdc3d4bb81e115ca73c8348b525b0d2338aa144667ed9431021412cd9b844cba29934aaacce873414ec11cb02e272d0ad81f767d33076721f13bf36020cf0b1fd0ecb078de1128beba0949ebece1a1f96e209dc36e4fffd36b673a7ddc1597ad4408e485c4adb2c873841249372523809e4312d0c7b3522ef983cac1e03935ff13a8e96ba681a62e40d3e70a7ff35866d3d9993f9e26a634c81b4e71380fcdd6f4e835f75a6409c7dc2c07410e6f87858c7b94c01c2e32f291769eacca71643b8b98a963df0a329bea4ed6397e8cd01a110ab361ac5bad1ccd840a6c8a6eaa001a9d7d87dc3318643571226c4dd2c2ac41fb
Added: BTW this only works because the signature scheme used for RSA in TLS1.2 and below, namely the scheme that was 'block type 1' in PKCS1v1 and now is RSASSA-PKCS1-v1_5 in PKCS1v2, is deterministic. Most digital signature schemes are not, including the RSA-PSS scheme used in TLS1.3, and you cannot check or test a signature by comparing it to another signature. You can only use the verification method provided by the scheme.
Method 2
is completely incorrect. \x00 etc is a notation used in source code for C and C++ (as you did in Method 1) and a few other languages like Java, JS/ES, and Python, as well as certain tools like the printf command in Unix and the awk program. But it does not work other places, and in particular does not work in files read by OpenSSL (at least for data; it might work in the config file, I'd have to check). Your method two (hashes and) signs the bytes 0x5c 0x78 0x30 0x30 0x5c 0x78 0x30 0x31 which represent the characters \ x 0 0 \ x 0 1 etc, not the bytes 0x00 0x01 etc, and unsurprisingly this is completely different and wrong.
I have sample code to fetch regions from Google Cloud API. This sample code works fine from my laptop (windows with OpenJDK 1.8 version). But the same code fails from kubernetes environment which has suse linux with OpenJDK 1.8 version.
From Suse linux side I get :
Exception in thread "main" java.io.IOException: Error getting access token for service account: Remote host closed connection during handshake
at com.google.auth.oauth2.ServiceAccountCredentials.refreshAccessToken(ServiceAccountCredentials.java:444)
at com.google.auth.oauth2.OAuth2Credentials.refresh(OAuth2Credentials.java:157)
at com.google.auth.oauth2.OAuth2Credentials.getRequestMetadata(OAuth2Credentials.java:145)
at com.google.auth.oauth2.ServiceAccountCredentials.getRequestMetadata(ServiceAccountCredentials.java:603)
at com.google.auth.http.HttpCredentialsAdapter.initialize(HttpCredentialsAdapter.java:91)
at com.google.api.client.http.HttpRequestFactory.buildRequest(HttpRequestFactory.java:91)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.buildHttpRequest(AbstractGoogleClientRequest.java:404)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:514)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:455)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:565)
at sample.program.gcp.vpvn.regionList(vpvn.java:85)
at sample.program.gcp.vpvn.main(vpvn.java:307)
Caused by: javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:994)
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1367)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1395)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1379)
at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:559)
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185)
at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1340)
at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1315)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:264)
at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:113)
at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:84)
at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1012)
at com.google.auth.oauth2.ServiceAccountCredentials.refreshAccessToken(ServiceAccountCredentials.java:441)
... 11 more
Caused by: java.io.EOFException: SSL peer shut down incorrectly
at sun.security.ssl.InputRecord.read(InputRecord.java:505)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:975)
... 23 more
When I enable SSL debug, I am not getting much details to troubleshoot this issue:
*** ClientHello, TLSv1.2
RandomCookie: GMT: 1616080171 bytes = { 119, 66, 219, 23, 171, 247, 221, 79, 45, 202, 181, 18, 229, 4, 65, 98, 207, 90, 0, 108, 43, 54, 80, 65, 39, 31, 49, 114 }
Session ID: {}
Cipher Suites: [TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_RSA_WITH_AES_256_CBC_SHA256, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384, TLS_DHE_RSA_WITH_AES_256_CBC_SHA256, TLS_DHE_DSS_WITH_AES_256_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_DSS_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256, TLS_EMPTY_RENEGOTIATION_INFO_SCSV]
Compression Methods: { 0 }
***
[write] MD5 and SHA1 hashes: len = 215
0000: 01 00 00 D3 03 03 60 53 6D 2B 77 42 DB 17 AB F7 ......`Sm+wB....
0010: DD 4F 2D CA B5 12 E5 04 41 62 CF 5A 00 6C 2B 36 .O-.....Ab.Z.l+6
0020: 50 41 27 1F 31 72 00 00 56 C0 24 C0 28 00 3D C0 PA'.1r..V.$.(.=.
0030: 26 C0 2A 00 6B 00 6A C0 0A C0 14 00 35 C0 05 C0 &.*.k.j.....5...
0040: 0F 00 39 00 38 C0 23 C0 27 00 3C C0 25 C0 29 00 ..9.8.#.'.<.%.).
0050: 67 00 40 C0 09 C0 13 00 2F C0 04 C0 0E 00 33 00 g.#...../.....3.
0060: 32 C0 2C C0 2B C0 30 00 9D C0 2E C0 32 00 9F 00 2.,.+.0.....2...
0070: A3 C0 2F 00 9C C0 2D C0 31 00 9E 00 A2 00 FF 01 ../...-.1.......
0080: 00 00 54 00 0A 00 08 00 06 00 17 00 18 00 19 00 ..T.............
0090: 0B 00 02 01 00 00 0D 00 1C 00 1A 06 03 06 01 05 ................
00A0: 03 05 01 04 03 04 01 04 02 03 03 03 01 03 02 02 ................
00B0: 03 02 01 02 02 00 17 00 00 00 00 00 1A 00 18 00 ................
00C0: 00 15 6F 61 75 74 68 32 2E 67 6F 6F 67 6C 65 61 ..oauth2.googlea
00D0: 70 69 73 2E 63 6F 6D pis.com
main, WRITE: TLSv1.2 Handshake, length = 215
[Raw write]: length = 220
0000: 16 03 03 00 D7 01 00 00 D3 03 03 60 53 6D 2B 77 ...........`Sm+w
0010: 42 DB 17 AB F7 DD 4F 2D CA B5 12 E5 04 41 62 CF B.....O-.....Ab.
0020: 5A 00 6C 2B 36 50 41 27 1F 31 72 00 00 56 C0 24 Z.l+6PA'.1r..V.$
0030: C0 28 00 3D C0 26 C0 2A 00 6B 00 6A C0 0A C0 14 .(.=.&.*.k.j....
0040: 00 35 C0 05 C0 0F 00 39 00 38 C0 23 C0 27 00 3C .5.....9.8.#.'.<
0050: C0 25 C0 29 00 67 00 40 C0 09 C0 13 00 2F C0 04 .%.).g.#...../..
0060: C0 0E 00 33 00 32 C0 2C C0 2B C0 30 00 9D C0 2E ...3.2.,.+.0....
0070: C0 32 00 9F 00 A3 C0 2F 00 9C C0 2D C0 31 00 9E .2...../...-.1..
0080: 00 A2 00 FF 01 00 00 54 00 0A 00 08 00 06 00 17 .......T........
0090: 00 18 00 19 00 0B 00 02 01 00 00 0D 00 1C 00 1A ................
00A0: 06 03 06 01 05 03 05 01 04 03 04 01 04 02 03 03 ................
00B0: 03 01 03 02 02 03 02 01 02 02 00 17 00 00 00 00 ................
00C0: 00 1A 00 18 00 00 15 6F 61 75 74 68 32 2E 67 6F .......oauth2.go
00D0: 6F 67 6C 65 61 70 69 73 2E 63 6F 6D ogleapis.com
main, received EOFException: error
main, handling exception: javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake
main, SEND TLSv1.2 ALERT: fatal, description = handshake_failure
Any hints on how to troubleshoot this issue?
Here with my sample code:
public static void main(String args[]) throws GeneralSecurityException, IOException {
Compute computeService = createComputeService();
Compute.Regions.List request = computeService.regions().list("imageagg-nonprod");
System.out.println("the list of regions for the selected project is \n");
RegionList response;
do {
response = request.execute();
if (response.getItems() == null) {
continue;
}
request.setPageToken(response.getNextPageToken());
} while (response.getNextPageToken() != null);
ArrayList regionNames = new ArrayList<String>();
HashMap<String, ArrayList<String>> ZoneList = new HashMap<>();
response.getItems().forEach(region -> {
ArrayList<String> zones = new ArrayList<String>();
regionNames.add(region.getName());
region.getZones().forEach(zone -> {
zones.add(Paths.get(URI.create(zone).getPath()).getFileName().toString());
});
ZoneList.put(region.getName(), zones);
});
System.out.println("list of region for selected project is \n");
regionNames.forEach(element -> {
System.out.println(element);
});
System.out.println("the names of regions and Zones for the selected Project is \n");
Set entries = ZoneList.entrySet();
Iterator it = entries.iterator();
while (it.hasNext()) {
Map.Entry pair = (Map.Entry) it.next();
System.out.println(pair.getKey() + " = " + pair.getValue());
}
machineList(ZoneList);
}
public static Compute createComputeService() throws IOException, GeneralSecurityException {
HttpTransport httpTransport = GoogleNetHttpTransport.newTrustedTransport();
String proxyHostOpt = "web-proxy.in.software.net";
int proxyPort = 8080;
JsonFactory jsonFactory = JacksonFactory.getDefaultInstance();
HttpTransport abc = new NetHttpTransport.Builder().trustCertificates(GoogleUtils.getCertificateTrustStore())
.setProxy(new Proxy(Proxy.Type.HTTP, InetSocketAddress.createUnresolved(proxyHostOpt, proxyPort))).build();
//GoogleCredential credential = GoogleCredential.getApplicationDefault(abc,jsonFactory);
List<String> scopes = new ArrayList<>();
//scopes.add("https://www.googleapis.com/auth/cloud-platform");
String jsonToken = "{\n" + " \"type\": \"service_account\",\n" + " \"project_id\": \"imageagg-nonprod\",\n" + " \"private_key_id\": \"99c871d2855b4d9388cc7a3a670a5764deb8c5e9\",\n" + " \"private_key\": \"-----BEGIN PRIVATE KEY-----\\nMIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQDh9k2JcCFrDZfm\\ng9DONfKe8xATwljEsW8FXMbPzU5JoXXsy1CYgkeW+eqXguQxFZM3HuI1W+mGBxgE\\n/K2P7XvJxylv7NajpgNmm4KGIh4hOpi+Sn3GVS31ftGM5A/CYKhRpr5uskr5PEin\\nDYxl0hUnfTodJCT+uxPxoCeN8aWuq5s+BapKKB8KVduUqmz3f8GL2Pc5wlm/YyOK\\nJYC781MAzLIFe8cLAVUJrVETqOtFTPCjy0yMGiUKxkyL20C11WFwfdD5ou0SD+6U\\nsT1YD/15KYh9GvV1E2XIPGzVtSHvU9h7FDRqOa+05QP3uDHegrAAib4PHA/A7KPD\\nBwkA6sW/AgMBAAECggEAHCPBtS9vIfdP5uecfcmvHMdVRbiquFgGZOsQYTmGmdnP\\nJz2MnGmBA9a8tc=\\n-----END PRIVATE KEY-----\\n\",\n" + " \"client_email\": \"315654350484-compute#developer.gserviceaccount.com\",\n" + " \"client_id\": \"112960668\",\n" + " \"auth_uri\": \"https://accounts.google.com/o/oauth2/auth\",\n" + " \"token_uri\": \"https://oauth2.googleapis.com/token\",\n" + " \"auth_provider_x509_cert_url\": \"https://www.googleapis.com/oauth2/v1/certs\",\n" + " \"client_x509_cert_url\": \"https://www.googleapis.com/robot/v1/metadata/x509/315654350484-compute%40developer.gserviceaccount.com\"\n" + "}";
ObjectMapper objectMapper = new ObjectMapper();
Map<String, Object> map
= objectMapper.readValue(jsonToken, new TypeReference<Map<String,Object>>(){});
scopes.add(ComputeScopes.COMPUTE);
scopes.add(ComputeScopes.CLOUD_PLATFORM);
//scopes.add(ComputeScopes.DEVSTORAGE_FULL_CONTROL);
GoogleCredentials credentials = GoogleCredentials.fromStream(IOUtils.toInputStream(jsonToken, StandardCharsets.UTF_8)).createScoped(scopes);
ServiceAccountCredentials serviceAccountCredentials = ServiceAccountCredentials.fromStream(IOUtils.toInputStream(jsonToken, StandardCharsets.UTF_8));
HttpRequestInitializer requestInitializer = new HttpCredentialsAdapter(credentials);
// Making call with credentials1 created with json string and proxy set as per requirements
return new Compute.Builder(abc, jsonFactory, requestInitializer).setApplicationName("hcmx").build();
}
My java version details:
java -version
openjdk version "11" 2018-09-25
OpenJDK Runtime Environment 18.9 (build 11+28)
OpenJDK 64-Bit Server VM 18.9 (build 11+28, mixed mode)
Environment where code is running:
[root#hcm-pool-centos76-3 ~]# uname -a
Linux hcm-pool-centos76-3 3.10.0-1062.9.1.el7.x86_64 #1 SMP Fri Dec 6 15:49:49 UTC 2
I have build application based on boost::asio. Sometimes I got this kind of core dump (not regullary). I tried investigate what's going on but I haven't more ideas to solve it.
In my point of view I think that could be some problem inside io service object - I mean maybe any bug? Should I update it ?
Anybody can explain what does memcpy() do in this case ? What is a reason of core ?
More details:
Platform . SunOS.
Boost - 1.49
/app/bin/executor_3'executor_dumpstack+0x13 [0x426645]
/app/bin/executor_3'signal_dumpstack+0x9d [0x426625]
/lib/amd64/libc.so.1'__sighndlr+0x6 [0xfffffd7fff224ea6]
/lib/amd64/libc.so.1'call_user_handler+0x2a4 [0xfffffd7fff217b5c]
/lib/amd64/libc.so.1'memcpy+0x1929 [0xfffffd7fff18a449] [Signal 11 (SEGV)]
/opt/lib/extralibs/exe_io.so'_ZNK5boost4_mfi3mf2Iv3GETRKNS_6system10error_codeEmE4callINS_10shared_ptrIS2_EES5_mEEvRT_PKvRT0_RT1_+0x8b [0xfffffd7ff64ff869]
/opt/lib/extralibs/exe_io.so'_ZNK5boost4_mfi3mf2Iv3GETRKNS_6system10error_codeEmEclINS_10shared_ptrIS2_EEEEvRT_S6_m+0x3c [0xfffffd7ff64fe47e]
/opt/lib/extralibs/exe_io.so'_ZN5boost3_bi5list3INS0_5valueINS_10shared_ptrI3GETEEEEPFNS_3argILi1EEEvEPFNS7_ILi2EEEvEEclINS_4_mfi3mf2IvS4_RKNS_6system10error_codeEmEENS0_5list2ISL_RKmEEEEvNS0_4typeIvEERT_RT0_i+0x72 [0xfffffd7ff64fce58]
/opt/lib/extralibs/exe_io.so'_ZN5boost3_bi6bind_tIvNS_4_mfi3mf2Iv3GETRKNS_6system10error_codeEmEENS0_5list3INS0_5valueINS_10shared_ptrIS4_EEEEPFNS_3argILi1EEEvEPFNSF_ILi2EEEvEEEEclIS6_mEEvRKT_RKT0_+0x43 [0xfffffd7ff64fbfa7]
/opt/lib/extralibs/exe_io.so'_ZN5boost4asio6detail17read_streambuf_opINS0_19basic_stream_socketINS0_2ip3tcpENS0_21stream_socket_serviceIS5_EEEESaIcENS1_18transfer_exactly_tENS_3_bi6bind_tIvNS_4_mfi3mf2Iv3GETRKNS_6system10error_codeEmEENSB_5list3INSB_5valueINS_10shared_ptrISF_EEEEPFNS_3argILi1EEEvEPFNSQ_ILi2EEEvEEEEEEclESJ_mi+0x16f [0xfffffd7ff64fa31b]
/opt/lib/extralibs/exe_io.so'_ZN5boost4asio6detail7binder2INS1_17read_streambuf_opINS0_19basic_stream_socketINS0_2ip3tcpENS0_21stream_socket_serviceIS6_EEEESaIcENS1_18transfer_exactly_tENS_3_bi6bind_tIvNS_4_mfi3mf2Iv3GETRKNS_6system10error_codeEmEENSC_5list3INSC_5valueINS_10shared_ptrISG_EEEEPFNS_3argILi1EEEvEPFNSR_ILi2EEEvEEEEEEESI_mEclEv+0x2d [0xfffffd7ff6503345]
/opt/lib/extralibs/exe_io.so'_ZN5boost4asio19asio_handler_invokeINS0_6detail7binder2INS2_17read_streambuf_opINS0_19basic_stream_socketINS0_2ip3tcpENS0_21stream_socket_serviceIS7_EEEESaIcENS2_18transfer_exactly_tENS_3_bi6bind_tIvNS_4_mfi3mf2Iv3GETRKNS_6system10error_codeEmEENSD_5list3INSD_5valueINS_10shared_ptrISH_EEEEPFNS_3argILi1EEEvEPFNSS_ILi2EEEvEEEEEEESJ_mEEEEvT_z+0x8e [0xfffffd7ff6502d76]
/opt/lib/extralibs/exe_io.so'_ZN33boost_asio_handler_invoke_helpers6invokeIN5boost4asio6detail7binder2INS3_17read_streambuf_opINS2_19basic_stream_socketINS2_2ip3tcpENS2_21stream_socket_serviceIS8_EEEESaIcENS3_18transfer_exactly_tENS1_3_bi6bind_tIvNS1_4_mfi3mf2Iv3GETRKNS1_6system10error_codeEmEENSE_5list3INSE_5valueINS1_10shared_ptrISI_EEEEPFNS1_3argILi1EEEvEPFNST_ILi2EEEvEEEEEEESK_mEES11_EEvRT_RT0_+0x3e [0xfffffd7ff650250a]
/opt/lib/extralibs/exe_io.so'_ZN5boost4asio6detail19asio_handler_invokeINS1_7binder2INS1_17read_streambuf_opINS0_19basic_stream_socketINS0_2ip3tcpENS0_21stream_socket_serviceIS7_EEEESaIcENS1_18transfer_exactly_tENS_3_bi6bind_tIvNS_4_mfi3mf2Iv3GETRKNS_6system10error_codeEmEENSD_5list3INSD_5valueINS_10shared_ptrISH_EEEEPFNS_3argILi1EEEvEPFNSS_ILi2EEEvEEEEEEESJ_mEESA_SB_SC_S10_EEvRT_PNS4_IT0_T1_T2_T3_EE+0x21 [0xfffffd7ff6501f61]
/opt/lib/extralibs/exe_io.so'_ZN33boost_asio_handler_invoke_helpers6invokeIN5boost4asio6detail7binder2INS3_17read_streambuf_opINS2_19basic_stream_socketINS2_2ip3tcpENS2_21stream_socket_serviceIS8_EEEESaIcENS3_18transfer_exactly_tENS1_3_bi6bind_tIvNS1_4_mfi3mf2Iv3GETRKNS1_6system10error_codeEmEENSE_5list3INSE_5valueINS1_10shared_ptrISI_EEEEPFNS1_3argILi1EEEvEPFNST_ILi2EEEvEEEEEEESK_mEES12_EEvRT_RT0_+0x25 [0xfffffd7ff65019a3]
/opt/lib/extralibs/exe_io.so'_ZN5boost4asio6detail23reactive_socket_recv_opINS0_17mutable_buffers_1ENS1_17read_streambuf_opINS0_19basic_stream_socketINS0_2ip3tcpENS0_21stream_socket_serviceIS7_EEEESaIcENS1_18transfer_exactly_tENS_3_bi6bind_tIvNS_4_mfi3mf2Iv3GETRKNS_6system10error_codeEmEENSD_5list3INSD_5valueINS_10shared_ptrISH_EEEEPFNS_3argILi1EEEvEPFNSS_ILi2EEEvEEEEEEEE11do_completeEPNS1_15task_io_serviceEPNS1_25task_io_service_operationESL_m+0xc4 [0xfffffd7ff65007cc]
/opt/lib/extralibs/exe_io.so'_ZN5boost4asio6detail25task_io_service_operation8completeERNS1_15task_io_serviceERKNS_6system10error_codeEm+0x32 [0xfffffd7ff6433dc8]
/opt/lib/extralibs/exe_io.so'_ZN5boost4asio6detail15task_io_service10do_run_oneERNS1_11scoped_lockINS1_11posix_mutexEEERNS2_11thread_infoERNS1_8op_queueINS1_25task_io_service_operationEEERKNS_6system10error_codeE+0x202 [0xfffffd7ff6433c9e]
/opt/lib/extralibs/exe_io.so'_ZN5boost4asio6detail15task_io_service3runERNS_6system10error_codeE+0xff [0xfffffd7ff6433925]
/opt/lib/extralibs/exe_io.so'_ZN5boost4asio10io_service3runEv+0x26 [0xfffffd7ff64335d6]
/opt/lib/extralibs/exe_io.so'_ZN6ZohaIO9RunIOEv+0x19 [0xfffffd7ff64335ad]
/opt/lib/extralibs/exe_io.so'_ZNK5boost4_mfi3mf0Iv6ZohaIOEclEPS2_+0x64 [0xfffffd7ff6440cee]
/opt/lib/extralibs/exe_io.so'_ZN5boost3_bi5list1INS0_5valueIP6ZohaIOEEEclINS_4_mfi3mf0IvS3_EENS0_5list0EEEvNS0_4typeIvEERT_RT0_i+0x41 [0xfffffd7ff6440c4d]
/opt/lib/extralibs/exe_io.so'_ZN5boost3_bi6bind_tIvNS_4_mfi3mf0Iv6ZohaIOEENS0_5list1INS0_5valueIPS4_EEEEEclEv+0x33 [0xfffffd7ff6440bfb]
/opt/lib/extralibs/exe_io.so'_ZN5boost6detail11thread_dataINS_3_bi6bind_tIvNS_4_mfi3mf0Iv6ZohaIOEENS2_5list1INS2_5valueIPS6_EEEEEEE3runEv+0x1c [0xfffffd7ff6440514]
/opt/csw/gxx/lib/amd64/libboost_thread.so.1.49.0'0xf655 [0xfffffd7ffa97f655]
/lib/amd64/libc.so.1'_thrp_setup+0xbc [0xfffffd7fff224b14]
/lib/amd64/libc.so.1'_lwp_start+0x0 [0xfffffd7fff224de0]
objdump output.
0000000000000000 <_ZNK5boost4_mfi3mf2Iv3GETRKNS_6system10error_codeEmE4callINS_10shared_ptrIS2_EES5_mEEvRT_PKvRT0_RT1_>:
template<class U, class B1, class B2> R call(U & u, T const *, B1 & b1, B2 & b2) const
{
BOOST_MEM_FN_RETURN (u.*f_)(b1, b2);
}
template<class U, class B1, class B2> R call(U & u, void const *, B1 & b1, B2 & b2) const
0: 55 push %rbp
1: 48 89 e5 mov %rsp,%rbp
4: 53 push %rbx
5: 48 83 ec 38 sub $0x38,%rsp
9: 48 89 7d e8 mov %rdi,-0x18(%rbp)
d: 48 89 75 e0 mov %rsi,-0x20(%rbp)
11: 48 89 55 d8 mov %rdx,-0x28(%rbp)
15: 48 89 4d d0 mov %rcx,-0x30(%rbp)
19: 4c 89 45 c8 mov %r8,-0x38(%rbp)
{
BOOST_MEM_FN_RETURN (get_pointer(u)->*f_)(b1, b2);
1d: 48 8b 45 e0 mov -0x20(%rbp),%rax
21: 48 89 c7 mov %rax,%rdi
24: e8 00 00 00 00 callq 29 <_ZNK5boost4_mfi3mf2Iv3GETRKNS_6system10error_codeEmE4callINS_10shared_ptrIS2_EES5_mEEvRT_PKvRT0_RT1_+0x29>
29: 48 89 c2 mov %rax,%rdx
2c: 48 8b 45 e8 mov -0x18(%rbp),%rax
30: 48 8b 00 mov (%rax),%rax
33: 83 e0 01 and $0x1,%eax
36: 84 c0 test %al,%al
38: 74 23 je 5d <_ZNK5boost4_mfi3mf2Iv3GETRKNS_6system10error_codeEmE4callINS_10shared_ptrIS2_EES5_mEEvRT_PKvRT0_RT1_+0x5d>
3a: 48 8b 45 e8 mov -0x18(%rbp),%rax
3e: 48 8b 40 08 mov 0x8(%rax),%rax
42: 48 8d 04 02 lea (%rdx,%rax,1),%rax
46: 48 8b 08 mov (%rax),%rcx
49: 48 8b 45 e8 mov -0x18(%rbp),%rax
4d: 48 8b 00 mov (%rax),%rax
50: 48 83 e8 01 sub $0x1,%rax
54: 48 8d 04 01 lea (%rcx,%rax,1),%rax
58: 48 8b 00 mov (%rax),%rax
5b: eb 07 jmp 64 <_ZNK5boost4_mfi3mf2Iv3GETRKNS_6system10error_codeEmE4callINS_10shared_ptrIS2_EES5_mEEvRT_PKvRT0_RT1_+0x64>
5d: 48 8b 45 e8 mov -0x18(%rbp),%rax
61: 48 8b 00 mov (%rax),%rax
64: 48 8b 4d c8 mov -0x38(%rbp),%rcx
68: 48 8b 19 mov (%rcx),%rbx
6b: 48 8b 4d e8 mov -0x18(%rbp),%rcx
6f: 48 8b 49 08 mov 0x8(%rcx),%rcx
73: 48 8d 3c 0a lea (%rdx,%rcx,1),%rdi
77: 48 8b 4d d0 mov -0x30(%rbp),%rcx
7b: 48 89 da mov %rbx,%rdx
7e: 48 89 ce mov %rcx,%rsi
Hint: use c++filt utility so your backtrace will become readable: cat backtrace | c++filt
Something happen to async_read handler on GET invoking. Maybe this object is destroyed at the time handler is invoked, or some mess with parameters. Cant say accurately without code, but something with read callback is what can be seen from this backtrace.
I am trying to write a small library for highly optimised x86-64 bit operation code and am fiddling with inline asm.
While testing this particular case has caught my attention:
unsigned long test = 0;
unsigned long bsr;
// bit test and set 39th bit
__asm__ ("btsq\t%1, %0 " : "+rm" (test) : "rJ" (39) );
// bit scan reverse (get most significant bit id)
__asm__ ("bsrq\t%1, %0" : "=r" (bsr) : "rm" (test) );
printf("test = %lu, bsr = %d\n", test, bsr);
compiles and runs fine in both gcc and icc, but when I inspect the assembly I get differences
gcc -S -fverbose-asm -std=gnu99 -O3
movq $0, -8(%rbp)
## InlineAsm Start
btsq $39, -8(%rbp)
## InlineAsm End
movq -8(%rbp), %rax
movq %rax, -16(%rbp)
## InlineAsm Start
bsrq -16(%rbp), %rdx
## InlineAsm End
movq -8(%rbp), %rsi
leaq L_.str(%rip), %rdi
xorb %al, %al
callq _printf
I am wondering why so complicated? I am writing high performance code in which the number of instructions is critical. I am especially wondering why gcc makes a copy of my variable test before passing it to the second inline asm?
Same code compiled with icc gives far better results:
xorl %esi, %esi # test = 0
movl $.L_2__STRING.0, %edi # has something to do with printf
orl $32832, (%rsp) # part of function initiation
xorl %eax, %eax # has something to do with printf
ldmxcsr (%rsp) # part of function initiation
btsq $39, %rsi #106.0
bsrq %rsi, %rdx #109.0
call printf #111.2
despite the fact that gcc decides to keep my variables on the stack rather then in registers, what I do not understand is why make a copy of test before passing it to the second asm?
If I put test in as an input/output variable in the second asm
__asm__ ("bsrq\t%1, %0" : "=r" (bsr) , "+rm" (test) );
then those lines disappear.
movq $0, -8(%rbp)
## InlineAsm Start
btsq $39, -8(%rbp)
## InlineAsm End
## InlineAsm Start
bsrq -8(%rbp), %rdx
## InlineAsm End
movq -8(%rbp), %rsi
leaq L_.str(%rip), %rdi
xorb %al, %al
callq _printf
Is this gcc screwed up optimisation or am I missing some vital compiler switches? I do have icc for my production system, but if I decide to distribute the source code at some point then it will have to be able to compile with gcc too.
compilers used:
gcc version 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2336.1.00)
icc Version 12.0.2
I've tried your example on Linux like this (making it "evil" by forcing a stack ref/loc for test through using &test in the printf:):#include <stdio.h>
int main(int argc, char **argv)
{
unsigned long test = 0;
unsigned long bsr;
// bit test and set 39th bit
asm ("btsq\t%1, %0 " : "+rm" (test) : "rJ" (39) );
// bit scan reverse (get most significant bit id)
asm ("bsrq\t%1, %0" : "=r" (bsr) : "rm" (test) );
printf("test = %lu, bsr = %d, &test = %p\n", test, bsr, &test);
return 0;
}
and compiled it with various versions of gcc -O3 ... to the following results:
code generated gcc version
================================================================================
400630: 48 83 ec 18 sub $0x18,%rsp 4.7.2,
400634: 31 c0 xor %eax,%eax 4.6.2,
400636: bf 50 07 40 00 mov $0x400750,%edi 4.4.6
40063b: 48 8d 4c 24 08 lea 0x8(%rsp),%rcx
400640: 48 0f ba e8 27 bts $0x27,%rax
400645: 48 89 44 24 08 mov %rax,0x8(%rsp)
40064a: 48 89 c6 mov %rax,%rsi
40064d: 48 0f bd d0 bsr %rax,%rdx
400651: 31 c0 xor %eax,%eax
400653: e8 68 fe ff ff callq 4004c0
[ ... ]
---------------------------------------------------------------------------------
4004f0: 48 83 ec 18 sub $0x18,%rsp 4.1
4004f4: 31 c0 xor %eax,%eax
4004f6: bf 28 06 40 00 mov $0x400628,%edi
4004fb: 48 8d 4c 24 10 lea 0x10(%rsp),%rcx
400500: 48 c7 44 24 10 00 00 00 00 movq $0x0,0x10(%rsp)
400509: 48 0f ba e8 27 bts $0x27,%rax
40050e: 48 89 44 24 10 mov %rax,0x10(%rsp)
400513: 48 89 c6 mov %rax,%rsi
400516: 48 0f bd d0 bsr %rax,%rdx
40051a: 31 c0 xor %eax,%eax
40051c: e8 c7 fe ff ff callq 4003e8
[ ... ]
---------------------------------------------------------------------------------
400500: 48 83 ec 08 sub $0x8,%rsp 3.4.5
400504: bf 30 06 40 00 mov $0x400630,%edi
400509: 31 c0 xor %eax,%eax
40050b: 48 c7 04 24 00 00 00 00 movq $0x0,(%rsp)
400513: 48 89 e1 mov %rsp,%rcx
400516: 48 0f ba 2c 24 27 btsq $0x27,(%rsp)
40051c: 48 8b 34 24 mov (%rsp),%rsi
400520: 48 0f bd 14 24 bsr (%rsp),%rdx
400525: e8 fe fe ff ff callq 400428
[ ... ]
---------------------------------------------------------------------------------
4004e0: 48 83 ec 08 sub $0x8,%rsp 3.2.3
4004e4: bf 10 06 40 00 mov $0x400610,%edi
4004e9: 31 c0 xor %eax,%eax
4004eb: 48 c7 04 24 00 00 00 00 movq $0x0,(%rsp)
4004f3: 48 0f ba 2c 24 27 btsq $0x27,(%rsp)
4004f9: 48 8b 34 24 mov (%rsp),%rsi
4004fd: 48 89 e1 mov %rsp,%rcx
400500: 48 0f bd 14 24 bsr (%rsp),%rdx
400505: e8 ee fe ff ff callq 4003f8
[ ... ]
and while there's a significant difference in the created code (including whether the bsr acceesses test as register or memory), none of the tested revs recreate the assembly that you've shown. I'd suspect a bug in the 4.2.x version you used on MacOSX, but then I don't have either your testcase nor that specific compiler version available.
Edit: The code above is obviously different in the sense that it forces test into the stack; if that is not done, then all "plain" gcc versions I've tested do a direct pair bts $39, %rsi / bsr %rsi, %rdx.
I have found, though, that clang creates different code there: 140: 50 push %rax
141: 48 c7 04 24 00 00 00 00 movq $0x0,(%rsp)
149: 31 f6 xor %esi,%esi
14b: 48 0f ba ee 27 bts $0x27,%rsi
150: 48 89 34 24 mov %rsi,(%rsp)
154: 48 0f bd d6 bsr %rsi,%rdx
158: bf 00 00 00 00 mov $0x0,%edi
15d: 30 c0 xor %al,%al
15f: e8 00 00 00 00 callq printf#plt>so the difference seems to be indeed between the code generators of clang/llvm and "gcc proper".