Related
I've searched StackOverflow trying to find a similar problem, but haven't come across it, so I am posting this question.
I am trying to write an C++ HTTPS client using Microsoft's SChannel libraries, and I'm getting stochastic errors with chunked message transfer. This issue only seems to occur on very long downloads -- short ones generally work OK. Most of the time the code works properly -- even for long downloads -- but occasionally the recv() command gracefully timesout, disconnecting my TLS session, and other times, I get an incomplete last packet. The stochastic errors appear to be the result of the different size chunks and encryption blocks the server is using to pass the data. I know I need to handle this variation, but while this would be easy to solve on an unencrypted HTTP connection, the encryption aspect is causing me problems.
First, the timeout problem, which occurs about 5% of the time I request large HTTP requests (about 10 MB of data from a single HTTP GET request).
The timeout is resulting because on the last chunk I have specified a bigger receive buffer than the data remaining on a blocking socket. The obvious fix to this is to only request exactly the number of bytes I need for the next chunk, and that is what I did. But for some reason, the amount received from each request is less than what I request, yet appears to be missing no data after decryption. I'm guessing this must be due to some compression in the data stream, but I don't know. IN any event, if it is using compression, I have no idea how to translate the size of the decrypted uncompressed byte stream into the size of compressed encrypted byte stream including the encryption headers and trailers to request the exact right number of bytes. Can anyone help me do that?
The alternative approach is for me to just look for two CR+LFs in a row, which would also signal the end of the HTTPS response. But because the data is encrypted, I can't figure out how to look byte by byte. SChannel's DecryptMessage() seems to do its decryptions in blocks, not byte by byte. Can anyone in this forum provide any advice on how to do byte-by-byte decryption to enable me to look for the end of the chunked output?
The second problem is DecryptMessage sometimes erroneously thinks it is done decrypting before I reach the actual end of the message. The resultant behavior is I go on to the next HTTP request, and I get the rest of the previous response where I am expecting to see the header of the new request.
The obvious solution to this is to check the contents of the decrypted message to see if we actually reached the end, and if not, try to receive more data before sending the next HTTP request. But when I do this, and try to decrypt, I get a decryption error message.
Any advice/help anyone can provide on a strategies would be appreciated. I've attached the relevant code sections for the read/decrypt process of the HTTP body -- I'm not including the header read and parsing because that is working without any problems.
do
{
// Note this receives large files OK, but I can't tell when I hit the end of the buffer, and this
// hangs. Need to consider a non-blocking socket?
// numBytesReceived = recv(windowsSocket, (char*)inputBuffer, inputBufSize, 0);
m_ErrorLog << "Next read size expected " << nextReadSize << endl;
numBytesReceived = recv(windowsSocket, (char*)inputBuffer, nextReadSize, 0);
m_ErrorLog << "NumBytesReceived = " << numBytesReceived << endl;
if (m_BinaryBufLen + numBytesReceived > m_BinaryBufAllocatedSize)
::EnlargeBinaryBuffer(m_BinaryBuffer,m_BinaryBufAllocatedSize,m_BinaryBufLen,numBytesReceived+1);
memcpy(m_BinaryBuffer+m_BinaryBufLen,inputBuffer,numBytesReceived);
m_BinaryBufLen += numBytesReceived;
lenStartDecryptedChunk = decryptedBodyLen;
do
{
// Decrypt the received data.
Buffers[0].pvBuffer = m_BinaryBuffer;
Buffers[0].cbBuffer = m_BinaryBufLen;
Buffers[0].BufferType = SECBUFFER_DATA; // Initial Type of the buffer 1
Buffers[1].BufferType = SECBUFFER_EMPTY; // Initial Type of the buffer 2
Buffers[2].BufferType = SECBUFFER_EMPTY; // Initial Type of the buffer 3
Buffers[3].BufferType = SECBUFFER_EMPTY; // Initial Type of the buffer 4
Message.ulVersion = SECBUFFER_VERSION; // Version number
Message.cBuffers = 4; // Number of buffers - must contain four SecBuffer structures.
Message.pBuffers = Buffers; // Pointer to array of buffers
scRet = m_pSSPI->DecryptMessage(phContext, &Message, 0, NULL);
if (scRet == SEC_E_INCOMPLETE_MESSAGE)
break;
if( scRet == SEC_I_CONTEXT_EXPIRED )
{
m_ErrorLog << "Server shut down connection before I finished reading" << endl;
m_ErrorLog << "# of Bytes Requested = " << nextReadSize << endl;
m_ErrorLog << "# of Bytes received = " << numBytesReceived << endl;
m_ErrorLog << "Decrypted data to this point = " << endl;
m_ErrorLog << decryptedBody << endl;
m_ErrorLog << "BinaryData just decrypted: " << endl;
m_ErrorLog << Buffers[0].pvBuffer << endl;
break; // Server signalled end of session
}
if( scRet != SEC_E_OK &&
scRet != SEC_I_RENEGOTIATE &&
scRet != SEC_I_CONTEXT_EXPIRED )
{
DisplaySECError((DWORD)scRet,errmsg);
m_ErrorLog << "CSISPDoc::ReadDecrypt(): " << "Failed to decrypt message--Error=" << errmsg;
if (decryptedBody)
m_ErrorLog << decryptedBody << endl;
return scRet;
}
// Locate data and (optional) extra buffers.
pDataBuffer = NULL;
pExtraBuffer = NULL;
for(i = 1; i < 4; i++)
{
if( pDataBuffer == NULL && Buffers[i].BufferType == SECBUFFER_DATA )
pDataBuffer = &Buffers[i];
if( pExtraBuffer == NULL && Buffers[i].BufferType == SECBUFFER_EXTRA )
pExtraBuffer = &Buffers[i];
}
// Display the decrypted data.
if(pDataBuffer)
{
length = pDataBuffer->cbBuffer;
if( length ) // check if last two chars are CR LF
{
buff = (PBYTE)pDataBuffer->pvBuffer; // printf( "n-2= %d, n-1= %d \n", buff[length-2], buff[length-1] );
if (decryptedBodyLen+length+1 > decryptedBodyAllocatedSize)
::EnlargeBuffer(decryptedBody,decryptedBodyAllocatedSize,decryptedBodyLen,length+1);
memcpy_s(decryptedBody+decryptedBodyLen,decryptedBodyAllocatedSize-decryptedBodyLen,buff,length);
decryptedBodyLen += length;
m_ErrorLog << buff << endl;
}
}
// Move any "extra" data to the input buffer -- this has not yet been decrypted.
if(pExtraBuffer)
{
MoveMemory(m_BinaryBuffer, pExtraBuffer->pvBuffer, pExtraBuffer->cbBuffer);
m_BinaryBufLen = pExtraBuffer->cbBuffer; // printf("inputStrLen= %d \n", inputStrLen);
}
}
while (pExtraBuffer);
if (decryptedBody)
{
if (incompletePacket)
p1 = decryptedBody + lenStartFragmentedPacket;
else
p1 = decryptedBody + lenStartDecryptedChunk;
p2 = p1;
pEndDecryptedBody = decryptedBody+decryptedBodyLen;
if (lastDecryptRes != SEC_E_INCOMPLETE_MESSAGE)
chunkSizeBlock = true;
do
{
while (p2 < pEndDecryptedBody && (*p2 != '\r' || *(p2+1) != '\n'))
p2++;
// if we're here, we probably found the end of the current line. The pattern we are
// reading is chunk length, chunk, chunk length, chunk,...,chunk lenth (==0)
if (*p2 == '\r' && *(p2+1) == '\n') // new line character -- found chunk size
{
if (chunkSizeBlock) // reading the size of the chunk
{
pStartHexNum = SkipWhiteSpace(p1,p2);
pEndHexNum = SkipWhiteSpaceBackwards(p1,p2);
chunkSize = HexCharToInt(pStartHexNum,pEndHexNum);
p2 += 2; // skip past the newline character
chunkSizeBlock = false;
if (!chunkSize) // chunk size of 0 means we're done
{
bulkReadDone = true;
p2 += 2; // skip past the final CR+LF
}
nextReadSize = chunkSize+8; // chunk + CR/LF + next chunk size (4 hex digits) + CR/LF + encryption header/trailer
}
else // copy the actual chunk
{
if (p2-p1 != chunkSize)
{
m_ErrorLog << "Warning: Actual chunk size of " << p2 - p1 << " != stated chunk size = " << chunkSize << endl;
}
else
{
// copy over the actual chunk data //
if (m_HTTPBodyLen + chunkSize > m_HTTPBodyAllocatedSize)
::EnlargeBuffer(m_HTTPBody,m_HTTPBodyAllocatedSize,m_HTTPBodyLen,chunkSize+1);
memcpy_s(m_HTTPBody+m_HTTPBodyLen,m_HTTPBodyAllocatedSize,p1,chunkSize);
m_HTTPBodyLen += chunkSize;
m_HTTPBody[m_HTTPBodyLen] = 0; // null-terminate
p2 += 2; // skip over chunk and end of line characters
chunkSizeBlock = true;
chunkSize = 0;
incompletePacket = false;
lenStartFragmentedPacket = 0;
}
}
p1 = p2; // move to start of next chunk field
}
else // got to end of encrypted body with no CR+LF found --> fragmeneted chunk. So we need to read and decrypt at least one more chunk
{
incompletePacket = true;
lenStartFragmentedPacket = p1-decryptedBody;
}
}
while (p2 < pEndDecryptedBody);
lastDecryptRes = scRet;
}
}
while (scRet == SEC_E_INCOMPLETE_MESSAGE && !bulkReadDone);
TLS does not support byte-by-byte decryption.
TLS 1.2 breaks its input into blocks of up to 16 kiB, then encrypts them into ciphertext blocks that are slightly larger due to the need for encryption IVs/nonces and integrity protection tags/MACs. It is not possible to decrypt a block until the entire block is available. You can find the full details at https://www.rfc-editor.org/rfc/rfc5246#section-6.2.
Since you're already able to decrypt the first few blocks (containing the headers), you should be able to read the HTTP length so that you at least know the plaintext length that you're expecting, which you can then compare to the number of bytes that you've decrypted from the stream. That won't tell you how many bytes of ciphertext you need, though -- you can get an upper bound on the size of a fragment by calling m_pSPPI->QueryContextAttributes() and then should read either at least that number of bytes or until end of stream before trying to decrypt.
Have you tried looking at other examples? http://www.coastrd.com/c-schannel-smtp appears to contain a detailed example of an SChannel-based TLS client.
I was finally able to figure this out. I fixed this by decrypting each TCP/IP packet as it came in to check for the CR+LF+CR+LF in the decrypted packet instead of what I had been doing -- trying to consolidate all of the encrypted packets into one buffer prior to decrypting it.
On the "hang" problem, what I thought was happening was that recv() wasn't returning because the amount of data actually received was smaller than my expected receive size. But what actually happened was I had actually received the entire transmission, but I didn't realize it. Thus, I was making additional recv() calls when there was actually no more data to receive. The fact that there was no more data to receive was what caused the connection to time out (causing a "hang").
The truncation problem was occurring because I couldn't detect the CR+LF+CR+LF sequence in the encrypted stream, and I erroneously thought SChannel returned SEC_E_OK on DecryptMessage() only when the entire response was processed.
Both problems were eliminated once I was able to detect the true end of the message by decrypting in piecemeal fashion vs. in bulk.
In order to figure this out, I had to completely restructure the sample SChannel code from www.coastRD.com. While the www.coastRD.com code was very helpful in general, it was written for SMTP transfers, not chunked HTTP encoding. In addition, the way it was written, it was hard to follow the logic for processing variations in how messages were received and processed. Lastly, I spent a lot of time "hacking" Schannel to understand how it behaves and which codes are returned under which conditions, because unfortunately none of that is discussed in any of the Microsoft documentation (that I've seen).
The first thing I needed to understand was how SChannel tries to decrypt a message. In Schannel, the 1st 13 bytes of an encrypted message are the encryption header, and the last 16 bytes are the encryption trailer. I still don't know what the trailer does, but I did realize that the encryption header is never actually encrypted/decrypted. The 1st 5 bytes are just the TLS record header for "application data" (hex code 0x17), followed by two bytes defining the TLS version used, followed by 2 bytes of the TLS record fragment size, followed by leading 0s and one byte which I still haven't figured out.
The reason this matters is that DecryptMessage() only works if the record type is "application data". For any other record type (such as a TLS handshake "finished message), DecryptMessage() won't even try to decrypt it-- it will just return a SEC_E_DECRYPT_FAILURE code.
In addition, I needed to understand that DecryptMessage() often can't decrypt the entire contents of the receive buffer in one pass when using chunked transfer encoding. In order to successfully process the entire contents of the receive buffer and the remainder of the server HTTPS response, I needed to understand two key return codes from DecryptMessage() -- SEC_E_OK and SEC_E_INCOMPLETE_MESSAGE.
When I received SEC_E_OK, it meant DecryptMessage() was able to successfully decrypt at least part of the receive buffer. When this occurred, the 1st 13 bytes (the encryption header) remained unchanged. However, the bytes immediately following the header were decrypted in-place, followed by the encryption trailer (which is also unchanged). Often, there will be additional encrypted data still in the receive buffer after the end of the encryption trailer, which is also unchanged.
Since I was using the SecBufferDesc output buffer structures and 4 SecBuffer structures described in www.coastRD.com's code, I needed to understand that these are not actually 4 separate buffers -- they are just pointers to different locations within the receive buffer. The first buffer is a pointer to the encryption header. The second buffer is a pointer to the beginning of the decrypted data. The 3rd buffer is a pointer to the beginning of the encryption trailer. Lastly, the 4th buffer is a pointer to the "extra" encrypted data that DecryptMessage() was not able to process on the last call.
Once I figured that out, I realized that I needed to copy the decrypted data (the pointer in the second buffer) into a separate buffer, because the receive buffer would probably be overwritten later.
If there was no "extra" data in the 4th buffer, I was done done for the moment -- but this was the exception rather than the rule.
If there was extra data (the usual case), I needed to move that data forward to the very beginning of the receive buffer, and I needed to call DecryptMessage() again. This decrypted the next chunk, and I appended that data to the data I already copied to the separate buffer, and repeated this process until there was either no more data left in the receive buffer to decrypt, or I received a SEC_E_INCOMPLETE_MESSAGE.
If I received a SEC_E_INCOMPLETE_MESSAGE, the data remaining in the receive buffer was unchanged. It wasn't decrypted because it was an incomplete encryption block. Thus, I needed to call recv() again to get more encrypted data from the server to complete the encryption block.
Once that occurred, I appended newly received data to the receive buffer. I appended it to the contents of the receive buffer vs. overwriting it because the latter approach would have overwritten the beginning of the encryption block, producing a SEC_E_DECRYPT_FAILURE message the next time I called DecryptMessage().
Once I appended this new block of data to the receive buffer, I repeated the steps above to decrypt the contents of the receive buffer, and continued to repeat this whole process until I got a SEC_E_OK message on the last chunk of data left in the receive buffer.
But I wasn't necessarily done yet -- there may still be data being sent by the server. Stopping at this point is what caused the truncation issue I had occasionally encountered.
So I now checked the last 4 bytes of the decrypted data to look for CR+LF+CR+LF. If I found that sequence, I knew I had received and decrypted a complete HTTPS response.
But if I hadn't, I needed to call recv() again and repeat the process above until I saw the CR+LF+FR+LF sequence at the end of the data.
Once I implemented this process, I was able to definitively identify the end of the encrypted HTTPS response, which prevented me from making an unnecessary recv() call when no data was remaining, preventing a "hang", as well as prematurely truncating the response.
I apologize for the long answer, but given the lack of documentation on SChannel and its functions like DecryptMessage(), I thought this description of what I learned might be helpful to others who may have also been struggling to use SChannel to process TLS HTTP responses.
Thank you again to user3553031 for trying to help me with this over 7 months ago -- those attempts helped me narrow down the problem.
I'm having trouble calculating the MAC of the finished message.The RFC gives the formula
HMAC_hash(MAC_write_secret, seq_num + TLSCompressed.type +
TLSCompressed.version + TLSCompressed.length +
TLSCompressed.fragment));
But the tlsCompressed(tlsplaintext in this case because no compression is used) does not contain version information:(hex dump)
14 00 00 0c 2c 93 e6 c5 d1 cb 44 12 bd a0 f9 2d
the first byte is the tlsplaintext.type, followed by uint24 length.
The full message, with the MAC and padding appended and before encryption is
1400000c2c93e6c5d1cb4412bda0f92dbc175a02daab04c6096da8d4736e7c3d251381b10b
I have tried to calculate the hmac with the following parameters(complying to the rfc) but it does not work:
uint64 seq_num
uint8 tlsplaintext.type
uint8 tlsplaintext.version_major
uint8 tlscompressed.version_minor
uint16 tlsplaintext.length
opaque tlsplaintext.fragment
I have also tried omitting the version and using uint24 length instead.no luck.
My hmac_hash() function cannot be the problem because it has worked thus far. I am also able to compute the verify_data and verify it.
Because this is the first message sent under the new connection state, the sequence number is 0.
So, what exactly are the parameters for the calculation of the MAC for the finished message?
Here's the relevant source from Forge (JS implementation of TLS 1.0):
The HMAC function:
var hmac_sha1 = function(key, seqNum, record) {
/* MAC is computed like so:
HMAC_hash(
key, seqNum +
TLSCompressed.type +
TLSCompressed.version +
TLSCompressed.length +
TLSCompressed.fragment)
*/
var hmac = forge.hmac.create();
hmac.start('SHA1', key);
var b = forge.util.createBuffer();
b.putInt32(seqNum[0]);
b.putInt32(seqNum[1]);
b.putByte(record.type);
b.putByte(record.version.major);
b.putByte(record.version.minor);
b.putInt16(record.length);
b.putBytes(record.fragment.bytes());
hmac.update(b.getBytes());
return hmac.digest().getBytes();
};
The function that creates the Finished record:
tls.createFinished = function(c) {
// generate verify_data
var b = forge.util.createBuffer();
b.putBuffer(c.session.md5.digest());
b.putBuffer(c.session.sha1.digest());
// TODO: determine prf function and verify length for TLS 1.2
var client = (c.entity === tls.ConnectionEnd.client);
var sp = c.session.sp;
var vdl = 12;
var prf = prf_TLS1;
var label = client ? 'client finished' : 'server finished';
b = prf(sp.master_secret, label, b.getBytes(), vdl);
// build record fragment
var rval = forge.util.createBuffer();
rval.putByte(tls.HandshakeType.finished);
rval.putInt24(b.length());
rval.putBuffer(b);
return rval;
};
The code to handle a Finished message is a bit lengthier and can be found here. I see that I have a comment in that code that sounds like it might be relevant to your problem:
// rewind to get full bytes for message so it can be manually
// digested below (special case for Finished messages because they
// must be digested *after* handling as opposed to all others)
Does this help you spot anything in your implementation?
Update 1
Per your comments, I wanted to clarify how TLSPlainText works. TLSPlainText is the main "record" for the TLS protocol. It is the "wrapper" or "envelope" for content-specific types of messages. It always looks like this:
struct {
ContentType type;
ProtocolVersion version;
uint16 length;
opaque fragment[TLSPlaintext.length];
} TLSPlaintext;
So it always has a version. A Finished message is a type of handshake message. All handshake messages have a content type of 22. A handshake message looks like this:
struct {
HandshakeType msg_type;
uint24 length;
body
} Handshake;
A Handshake message is yet another envelope/wrapper for other messages, like the Finished message. In this case, the body will be a Finished message (HandshakeType 20), which looks like this:
struct {
opaque verify_data[12];
} Finished;
To actually send a Finished message, you have to wrap it up in a Handshake message envelope, and then like any other message, you have to wrap it up in a TLS record (TLSPlainText). The ultimate result looks/represents something like this:
struct {
ContentType type=22;
ProtocolVersion version=<major, minor>;
uint16 length=<length of fragment>;
opaque fragment=<struct {
HandshakeType msg_type=20;
uint24 length=<length of finished message>;
body=<struct {
opaque verify_data[12]>;
} Finished>
} Handshake>
} TLSPlainText;
Then, before transport, the record may be altered. You can think of these alterations as operations that take a record and transform its fragment (and fragment length). The first operation compresses the fragment. After compression you compute the MAC, as described above and then append that to the fragment. Then you encrypt the fragment (adding the appropriate padding if using a block cipher) and replace it with the ciphered result. So, when you're finished, you've still got a record with a type, version, length, and fragment, but the fragment is encrypted.
So, just so we're clear, when you're computing the MAC for the Finished message, imagine passing in the above TLSPlainText (assuming there's no compression as you indicated) to a function. This function takes this TLSPlainText record, which has properties for type, version, length, and fragment. The HMAC function above is run on the record. The HMAC key and sequence number (which is 0 here) are provided via the session state. Therefore, you can see that everything the HMAC function needs is available.
In any case, hopefully this better explains how the protocol works and that will maybe reveal what's going wrong with your implementation.
Hi guys I needed simple RSA Encryption Decryption.
I tried the code examples on Apple developer guide, it works perfectly for small amount of text but the example code doesn't cater for situations of large encrypted data.
Take note of the comment that it is suggesting us to "split the data up into blocks equal to plainBufferSize":
- (NSData*)decryptedDataFromData:(NSData*)data usingKey:(SecKeyRef)key
{
OSStatus status = noErr;
size_t cipherBufferSize = [data length];
uint8_t *cipherBuffer = (uint8_t *)[data bytes];
size_t plainBufferSize;
uint8_t *plainBuffer;
// Allocate the buffer
plainBufferSize = SecKeyGetBlockSize(key);
plainBuffer = malloc(plainBufferSize);
if (plainBufferSize < cipherBufferSize) {
// Ordinarily, you would split the data up into blocks
// equal to plainBufferSize, with the last block being
// shorter. For simplicity, this example assumes that
// the data is short enough to fit.
printf("Could not decrypt. Packet too large.\n");
return nil;
}
// Error handling
status = SecKeyDecrypt(key,
kSecPaddingPKCS1,
cipherBuffer,
cipherBufferSize,
plainBuffer,
&plainBufferSize
); // 3
// Error handling
// Store or display the decrypted text
if(key) CFRelease(key);
NSData *decrypted = [NSData dataWithBytes:(const void *)plainBuffer length:plainBufferSize];
return decrypted;
}
Any clues on how should I modify this method so that it will split the data in blocks to handle large amount of data?
According to RFC3447 RSAES-PKCS1-v1_5 encryption scheme you are using can operate on messages of length up to k - 11 octets (k is the octet length of the RSA modulus) so if you are using 2048-bit RSA key then maximum length of the plain data to be encrypted is 245 bytes. So you will need to split plain data to the chunks of this size and then encrypt each of them individually but this is rather rare and slow solution. It is much better (and also pretty common) to generate symmetric AES key, encrypt large data using AES algorithm and then encrypt small AES key with RSA key.
I have set up a method in vb.net and in xcode for encrypting a string using as far as i can tell the same parameters for an AES encryption.
I've looked all over the place but cannot find information on whether they use the same encryption algorithm and settings.
this is the vb.net code:
Dim encryptAES As New AesCryptoServiceProvider()
Dim encoding As New UTF8Encoding()
Dim encryptor As ICryptoTransform
encryptAES.Key = encoding.GetBytes("12345678901234567890123456789032")
encryptAES.IV = encoding.GetBytes("1234567890123416")
encryptAES.Mode = CipherMode.CBC
encryptAES.Padding = PaddingMode.PKCS7
encryptor = encryptAES.CreateEncryptor
Dim input As Byte() = encoding.GetBytes("hello")
Dim result = encryptor.TransformFinalBlock(input, 0, input.Length)
Dim hex As String = ""
For i As Integer = 0 To result.Length - 1
hex = hex & result(i).ToString("X2")
Next
ConsoleWrite(hex)
which matches the output of this php encryption:
<?php
$cipher = mcrypt_module_open(MCRYPT_RIJNDAEL_128, '', MCRYPT_MODE_CBC, '');
$key256 = '12345678901234567890123456789032';
$iv = '1234567890123416';
$cleartext = 'hello';
printf("plainText: %s\n\n",$cleartext);
if (mcrypt_generic_init($cipher, $key256, $iv) != -1)
{
$cipherText = mcrypt_generic($cipher,$cleartext );
mcrypt_generic_deinit($cipher);
printf("256-bit encrypted result:\n%s\n\n",bin2hex($cipherText));
}
?>
And here's the mac code:
IV = [#"1234567890123416" dataUsingEncoding:NSUTF8StringEncoding];
NSLog([IV base64EncodingWithLineLength:0]);
NSData * localKey = [#"12345678901234567890123456789032" dataUsingEncoding:NSUTF8StringEncoding];
NSLog([localKey base64EncodingWithLineLength:0]);
NSLog(#"encode plaintext:");
NSLog([self.data base64EncodingWithLineLength:0]);
CCCryptorStatus result = CCCrypt( kCCEncrypt, kCCAlgorithmAES128, kCCOptionPKCS7Padding,
localKey, kCCKeySizeAES256,
IV, //initialization vector
[self.data mutableBytes], [self.data length], //input
buffer, bufferSize, //output
&numBytesEncrypted );
self.data = [self.data initWithBytes:buffer length:numBytesEncrypted];
NSLog(#"encode end result:");
NSLog([self.data base64EncodingWithLineLength:0]);
So as far as I can tell the mac code must be incorrect as it returns a different value (the same length).
Here are the debug dumps from both the pac and mac versions.
pc (values in base64):
key: MTIzNDU2Nzg5MDEyMzQ1Njc4OTAxMjM0NTY3ODkwMzI=
iv: MTIzNDU2Nzg5MDEyMzQxNg==
plaintext: aGVsbG8=
encrypted: oiUIdi9StezV93+nXctCKw==
mac (values in base 64):
key: MTIzNDU2Nzg5MDEyMzQ1Njc4OTAxMjM0NTY3ODkwMzI=
iv: MTIzNDU2Nzg5MDEyMzQxNg==
plaintext: aGVsbG8=
encrypted: rU5MgfNIjx7zqD5Cdh2mlA==
I also had a look at endianness here is the pc encryped base64 with both endianness and neither match the mac version:
oiUIdi9StezV93+nXctCKw==
K0LLXad/99XstVIvdgglog==
I've now run a test where I've set the key to be 32 0's and the IV to be 16 0's (so endianness should hopefully not matter) and with two more tests heres the results from mac:
BfG2gjzkooiJCXgIQJKyZg== (iOS 5 on device)
n9KJXqThiuKyrNPjo7V7PA== (iOS 5 in simulator)
cCH116bIlLeUOxNDwFt2rg== (iOS 4.3 in simulator)
WTH? all parameters are identical, only change is which version to compile for and each encryption is different...
and the pc result:
wjXeJNI54DzI43fGBPymew==
When inputting the mac encrypted text into the pc to decrypt, I get a 'Padding is invalid and cannot be removed.' error.
Any information anyone could provide would be much appreciated.
Both the mac and pc versions involve first creating a byte array from the sample text, and it is this byte array that is then passed to the encryption algorithm. I would start by comparing those byte arrays to make sure you are encrypting the same data. It looks like in both cases you have utf8 and AES128, but it never hurts to be sure.
Next, the result of the encryption algorithm is also a byte array, which must then be re-encoded as a string. Looking at your results, I see not just two different values, but two completely different kinds of characters. That tells me that the re-encoding may be off. Again, you want to compare the byte arrays.
In both cases, to ensure an accurate compare, I would base64 encode the arrays.
How can I use OpenSSL's ECC support to encrypt or decrypt a text string? I am able to generate ECC private/public keys using OpenSSL APIs, but I don't know how to encrypt plain text using those keys.
Since its so hard to find examples showing how to use ECC to encrypt data I thought I'd post some code for others to use. For the complete listing, check out my openssl-dev posting:
http://www.mail-archive.com/openssl-dev#openssl.org/msg28042.html
Basically its a flushed out usable version of how to use ECDH to secure a block of data. ECDH is used to generate a shared secret. The shared secret is then hashed using SHA 512. The resulting 512 bits are split up, with 256 serving as the key to the symmetric cipher (AES 256 in my example) and the other 256 bits used as the key for the HMAC. My implementation is loosely based on the ECIES standard outlined by SECG working group.
The key functions are ecies_encrypt() which accepts the public key in hex form and returns the encrypted data:
secure_t * ecies_encrypt(char *key, unsigned char *data, size_t length) {
void *body;
HMAC_CTX hmac;
int body_length;
secure_t *cryptex;
EVP_CIPHER_CTX cipher;
unsigned int mac_length;
EC_KEY *user, *ephemeral;
size_t envelope_length, block_length, key_length;
unsigned char envelope_key[SHA512_DIGEST_LENGTH], iv[EVP_MAX_IV_LENGTH], block[EVP_MAX_BLOCK_LENGTH];
// Simple sanity check.
if (!key || !data || !length) {
printf("Invalid parameters passed in.\n");
return NULL;
}
// Make sure we are generating enough key material for the symmetric ciphers.
if ((key_length = EVP_CIPHER_key_length(ECIES_CIPHER)) * 2 > SHA512_DIGEST_LENGTH) {
printf("The key derivation method will not produce enough envelope key material for the chosen ciphers. {envelope = %i / required = %zu}", SHA512_DIGEST_LENGTH / 8,
(key_length * 2) / 8);
return NULL;
}
// Convert the user's public key from hex into a full EC_KEY structure.
if (!(user = ecies_key_create_public_hex(key))) {
printf("Invalid public key provided.\n");
return NULL;
}
// Create the ephemeral key used specifically for this block of data.
else if (!(ephemeral = ecies_key_create())) {
printf("An error occurred while trying to generate the ephemeral key.\n");
EC_KEY_free(user);
return NULL;
}
// Use the intersection of the provided keys to generate the envelope data used by the ciphers below. The ecies_key_derivation() function uses
// SHA 512 to ensure we have a sufficient amount of envelope key material and that the material created is sufficiently secure.
else if (ECDH_compute_key(envelope_key, SHA512_DIGEST_LENGTH, EC_KEY_get0_public_key(user), ephemeral, ecies_key_derivation) != SHA512_DIGEST_LENGTH) {
printf("An error occurred while trying to compute the envelope key. {error = %s}\n", ERR_error_string(ERR_get_error(), NULL));
EC_KEY_free(ephemeral);
EC_KEY_free(user);
return NULL;
}
// Determine the envelope and block lengths so we can allocate a buffer for the result.
else if ((block_length = EVP_CIPHER_block_size(ECIES_CIPHER)) == 0 || block_length > EVP_MAX_BLOCK_LENGTH || (envelope_length = EC_POINT_point2oct(EC_KEY_get0_group(
ephemeral), EC_KEY_get0_public_key(ephemeral), POINT_CONVERSION_COMPRESSED, NULL, 0, NULL)) == 0) {
printf("Invalid block or envelope length. {block = %zu / envelope = %zu}\n", block_length, envelope_length);
EC_KEY_free(ephemeral);
EC_KEY_free(user);
return NULL;
}
// We use a conditional to pad the length if the input buffer is not evenly divisible by the block size.
else if (!(cryptex = secure_alloc(envelope_length, EVP_MD_size(ECIES_HASHER), length, length + (length % block_length ? (block_length - (length % block_length)) : 0)))) {
printf("Unable to allocate a secure_t buffer to hold the encrypted result.\n");
EC_KEY_free(ephemeral);
EC_KEY_free(user);
return NULL;
}
// Store the public key portion of the ephemeral key.
else if (EC_POINT_point2oct(EC_KEY_get0_group(ephemeral), EC_KEY_get0_public_key(ephemeral), POINT_CONVERSION_COMPRESSED, secure_key_data(cryptex), envelope_length,
NULL) != envelope_length) {
printf("An error occurred while trying to record the public portion of the envelope key. {error = %s}\n", ERR_error_string(ERR_get_error(), NULL));
EC_KEY_free(ephemeral);
EC_KEY_free(user);
secure_free(cryptex);
return NULL;
}
// The envelope key has been stored so we no longer need to keep the keys around.
EC_KEY_free(ephemeral);
EC_KEY_free(user);
// For now we use an empty initialization vector.
memset(iv, 0, EVP_MAX_IV_LENGTH);
// Setup the cipher context, the body length, and store a pointer to the body buffer location.
EVP_CIPHER_CTX_init(&cipher);
body = secure_body_data(cryptex);
body_length = secure_body_length(cryptex);
// Initialize the cipher with the envelope key.
if (EVP_EncryptInit_ex(&cipher, ECIES_CIPHER, NULL, envelope_key, iv) != 1 || EVP_CIPHER_CTX_set_padding(&cipher, 0) != 1 || EVP_EncryptUpdate(&cipher, body,
&body_length, data, length - (length % block_length)) != 1) {
printf("An error occurred while trying to secure the data using the chosen symmetric cipher. {error = %s}\n", ERR_error_string(ERR_get_error(), NULL));
EVP_CIPHER_CTX_cleanup(&cipher);
secure_free(cryptex);
return NULL;
}
// Check whether all of the data was encrypted. If they don't match up, we either have a partial block remaining, or an error occurred.
else if (body_length != length) {
// Make sure all that remains is a partial block, and their wasn't an error.
if (length - body_length >= block_length) {
printf("Unable to secure the data using the chosen symmetric cipher. {error = %s}\n", ERR_error_string(ERR_get_error(), NULL));
EVP_CIPHER_CTX_cleanup(&cipher);
secure_free(cryptex);
return NULL;
}
// Copy the remaining data into our partial block buffer. The memset() call ensures any extra bytes will be zero'ed out.
memset(block, 0, EVP_MAX_BLOCK_LENGTH);
memcpy(block, data + body_length, length - body_length);
// Advance the body pointer to the location of the remaining space, and calculate just how much room is still available.
body += body_length;
if ((body_length = secure_body_length(cryptex) - body_length) < 0) {
printf("The symmetric cipher overflowed!\n");
EVP_CIPHER_CTX_cleanup(&cipher);
secure_free(cryptex);
return NULL;
}
// Pass the final partially filled data block into the cipher as a complete block. The padding will be removed during the decryption process.
else if (EVP_EncryptUpdate(&cipher, body, &body_length, block, block_length) != 1) {
printf("Unable to secure the data using the chosen symmetric cipher. {error = %s}\n", ERR_error_string(ERR_get_error(), NULL));
EVP_CIPHER_CTX_cleanup(&cipher);
secure_free(cryptex);
return NULL;
}
}
// Advance the pointer, then use pointer arithmetic to calculate how much of the body buffer has been used. The complex logic is needed so that we get
// the correct status regardless of whether there was a partial data block.
body += body_length;
if ((body_length = secure_body_length(cryptex) - (body - secure_body_data(cryptex))) < 0) {
printf("The symmetric cipher overflowed!\n");
EVP_CIPHER_CTX_cleanup(&cipher);
secure_free(cryptex);
return NULL;
}
else if (EVP_EncryptFinal_ex(&cipher, body, &body_length) != 1) {
printf("Unable to secure the data using the chosen symmetric cipher. {error = %s}\n", ERR_error_string(ERR_get_error(), NULL));
EVP_CIPHER_CTX_cleanup(&cipher);
secure_free(cryptex);
return NULL;
}
EVP_CIPHER_CTX_cleanup(&cipher);
// Generate an authenticated hash which can be used to validate the data during decryption.
HMAC_CTX_init(&hmac);
mac_length = secure_mac_length(cryptex);
// At the moment we are generating the hash using encrypted data. At some point we may want to validate the original text instead.
if (HMAC_Init_ex(&hmac, envelope_key + key_length, key_length, ECIES_HASHER, NULL) != 1 || HMAC_Update(&hmac, secure_body_data(cryptex), secure_body_length(cryptex))
!= 1 || HMAC_Final(&hmac, secure_mac_data(cryptex), &mac_length) != 1) {
printf("Unable to generate a data authentication code. {error = %s}\n", ERR_error_string(ERR_get_error(), NULL));
HMAC_CTX_cleanup(&hmac);
secure_free(cryptex);
return NULL;
}
HMAC_CTX_cleanup(&hmac);
return cryptex;
}
And ecies_decrypt() which takes the private key, again in hex form, and decrypts the previously secured buffer:
unsigned char * ecies_decrypt(char *key, secure_t *cryptex, size_t *length) {
HMAC_CTX hmac;
size_t key_length;
int output_length;
EVP_CIPHER_CTX cipher;
EC_KEY *user, *ephemeral;
unsigned int mac_length = EVP_MAX_MD_SIZE;
unsigned char envelope_key[SHA512_DIGEST_LENGTH], iv[EVP_MAX_IV_LENGTH], md[EVP_MAX_MD_SIZE], *block, *output;
// Simple sanity check.
if (!key || !cryptex || !length) {
printf("Invalid parameters passed in.\n");
return NULL;
}
// Make sure we are generating enough key material for the symmetric ciphers.
else if ((key_length = EVP_CIPHER_key_length(ECIES_CIPHER)) * 2 > SHA512_DIGEST_LENGTH) {
printf("The key derivation method will not produce enough envelope key material for the chosen ciphers. {envelope = %i / required = %zu}", SHA512_DIGEST_LENGTH / 8,
(key_length * 2) / 8);
return NULL;
}
// Convert the user's public key from hex into a full EC_KEY structure.
else if (!(user = ecies_key_create_private_hex(key))) {
printf("Invalid private key provided.\n");
return NULL;
}
// Create the ephemeral key used specifically for this block of data.
else if (!(ephemeral = ecies_key_create_public_octets(secure_key_data(cryptex), secure_key_length(cryptex)))) {
printf("An error occurred while trying to recreate the ephemeral key.\n");
EC_KEY_free(user);
return NULL;
}
// Use the intersection of the provided keys to generate the envelope data used by the ciphers below. The ecies_key_derivation() function uses
// SHA 512 to ensure we have a sufficient amount of envelope key material and that the material created is sufficiently secure.
else if (ECDH_compute_key(envelope_key, SHA512_DIGEST_LENGTH, EC_KEY_get0_public_key(ephemeral), user, ecies_key_derivation) != SHA512_DIGEST_LENGTH) {
printf("An error occurred while trying to compute the envelope key. {error = %s}\n", ERR_error_string(ERR_get_error(), NULL));
EC_KEY_free(ephemeral);
EC_KEY_free(user);
return NULL;
}
// The envelope key material has been extracted, so we no longer need the user and ephemeral keys.
EC_KEY_free(ephemeral);
EC_KEY_free(user);
// Use the authenticated hash of the ciphered data to ensure it was not modified after being encrypted.
HMAC_CTX_init(&hmac);
// At the moment we are generating the hash using encrypted data. At some point we may want to validate the original text instead.
if (HMAC_Init_ex(&hmac, envelope_key + key_length, key_length, ECIES_HASHER, NULL) != 1 || HMAC_Update(&hmac, secure_body_data(cryptex), secure_body_length(cryptex))
!= 1 || HMAC_Final(&hmac, md, &mac_length) != 1) {
printf("Unable to generate the authentication code needed for validation. {error = %s}\n", ERR_error_string(ERR_get_error(), NULL));
HMAC_CTX_cleanup(&hmac);
return NULL;
}
HMAC_CTX_cleanup(&hmac);
// We can use the generated hash to ensure the encrypted data was not altered after being encrypted.
if (mac_length != secure_mac_length(cryptex) || memcmp(md, secure_mac_data(cryptex), mac_length)) {
printf("The authentication code was invalid! The ciphered data has been corrupted!\n");
return NULL;
}
// Create a buffer to hold the result.
output_length = secure_body_length(cryptex);
if (!(block = output = malloc(output_length + 1))) {
printf("An error occurred while trying to allocate memory for the decrypted data.\n");
return NULL;
}
// For now we use an empty initialization vector. We also clear out the result buffer just to be on the safe side.
memset(iv, 0, EVP_MAX_IV_LENGTH);
memset(output, 0, output_length + 1);
EVP_CIPHER_CTX_init(&cipher);
// Decrypt the data using the chosen symmetric cipher.
if (EVP_DecryptInit_ex(&cipher, ECIES_CIPHER, NULL, envelope_key, iv) != 1 || EVP_CIPHER_CTX_set_padding(&cipher, 0) != 1 || EVP_DecryptUpdate(&cipher, block,
&output_length, secure_body_data(cryptex), secure_body_length(cryptex)) != 1) {
printf("Unable to decrypt the data using the chosen symmetric cipher. {error = %s}\n", ERR_error_string(ERR_get_error(), NULL));
EVP_CIPHER_CTX_cleanup(&cipher);
free(output);
return NULL;
}
block += output_length;
if ((output_length = secure_body_length(cryptex) - output_length) != 0) {
printf("The symmetric cipher failed to properly decrypt the correct amount of data!\n");
EVP_CIPHER_CTX_cleanup(&cipher);
free(output);
return NULL;
}
if (EVP_DecryptFinal_ex(&cipher, block, &output_length) != 1) {
printf("Unable to decrypt the data using the chosen symmetric cipher. {error = %s}\n", ERR_error_string(ERR_get_error(), NULL));
EVP_CIPHER_CTX_cleanup(&cipher);
free(output);
return NULL;
}
EVP_CIPHER_CTX_cleanup(&cipher);
*length = secure_orig_length(cryptex);
return output;
}
I'm posting this because I personally couldn't find any other examples of how to secure files using ECC and the OpenSSL library. That said its worth mentioning alternatives that don't use OpenSSL. One is seccure which follows a pattern similar to my example, only it relies libgcrypt. Since libgcrypt doesn't provide all of the underlying ECC functions needed, the seccure program fills in the gaps and implements the ECC logic missing from libgcrypt.
Another program worth looking at is SKS, which uses a similar ECC based encryption process as the example above, but doesn't have any external dependencies (so all the ECC code is right there for you to look at).
ECC itself doesn't really define any encryption/decryption operations - algorithms built on elliptic curves do.
One example is Elliptic-Curve Diffie-Hellman. You could encrypt a message using ECDH by:
Generating an ephemeral EC key.
Using that key and the public key of the recipient, generate a secret using ECDH.
Use that secret as a key to encrypt the message with a symmetric cipher, like AES.
Transmit the encrypted message and the ephemeral public key generated in step 1.
To decrypt:
Load the ephemeral public key from the message.
Use that public key together with your recipient private key to generate a secret using ECDH.
Use that secret as a key to decrypt the message with the symmetric cipher.
EDIT: The following is the basic idea to generate a secret using ECDH. First we need to define a key derivation function - this one uses the SHA1 hash.
void *KDF1_SHA1(const void *in, size_t inlen, void *out, size_t *outlen)
{
if (*outlen < SHA_DIGEST_LENGTH)
return NULL;
else
*outlen = SHA_DIGEST_LENGTH;
return SHA1(in, inlen, out);
}
This is the ECDH code for the sender side. It assumes that the recipient's public key is already in "recip_key", and you have verified it with EC_KEY_check_key(). It also omits much important error checking, for the sake of brevity, which you will definitely want to include in production code.
EC_KEY *ephemeral_key = NULL;
const EC_GROUP *group = NULL;
unsigned char buf[SHA_DIGEST_LENGTH] = { 0 };
group = EC_KEY_get0_group(recip_key);
ephemeral_key = EC_KEY_new();
EC_KEY_set_group(ephemeral_key, group);
EC_KEY_generate_key(ephemeral_key);
ECDH_compute_key(buf, sizeof buf, EC_KEY_get0_public_key(recip_key), ephemeral_key, KDF1_SHA1);
After this the buffer 'buf' contains 20 bytes of material you can use for keying. This abbreviated example is based on the code in "ecdhtest.c" in the openssl source distribution - I suggest taking a look at it.
You will want to send the public key portion of ephemeral_key with the encrypted message, and securely discard the private key portion. A MAC over the data is also a good idea, and if you need more than 20 bytes of keying material a longer hash is probably in order.
The recipient does something similar, except that its private key already exists (since the sender had to know the corresponding public key beforehand), and the public key is recieved from the sender.