How to do ECDHE handshake without exportable private key - ssl

Question

I'm building an OpenSSL engine that implements ECDSA_METHOD, which includes signature creation and signature verification functions. Since the only usage of ECDHE private key is related to signature creation, having the key exported from the engine and presenting it anywhere else is not required.
However, if I don't supply the private key to SSL_Context through SSL_set_private_key function SSL handshake fails with the error below:
error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure
I've also tried to provide a mock key (one that is not related to a public key in the cert) to SSL_set_private_key function, but this function does verify if private/public keys match and throws an error about bad certificate if they don't.
It looks like openssl allows by-passing this validation in some cases, e.g. this is what I found in ssl/ssl_rsa.c
#ifndef OPENSSL_NO_RSA
/*
* Don't check the public/private key, this is mostly for smart
* cards.
*/
if ((pkey->type == EVP_PKEY_RSA) &&
(RSA_flags(pkey->pkey.rsa) & RSA_METHOD_FLAG_NO_CHECK)) ;
else
#endif
if (!X509_check_private_key(c->pkeys[i].x509, pkey)) {
X509_free(c->pkeys[i].x509);
c->pkeys[i].x509 = NULL;
return 0;
}
I think, I need something similar for an EC key, but I didn't find it anywhere. Any other solutions are appreciated as well.

Answer 1

Any other solutions are appreciated as well.
This might not be the only option you have, but I think that you can achieve what you are looking for by creating your own EVP_PKEY_METHOD and implementing its functions as required. That way, you can store a handle to your own, for example, smart card based key and then invoke the proper sign methods at the right moment. You have to set the proper methods with the EVP_PKEY_meth_set_Xyz() functions, like EVP_PKEY_meth_set_sign(<yourSigningFunction>). For example, if you were using the Windows crypto API, you would have to invoke NCryptSignHash() from your signing function. That way, you do not have to export the private key from the Windows key store to obtain a signature.
I have done this before and the only big thing I ran into (apart from lack of documentation and examples) was a missing key store functionality at the EVP level. There seems to be some work in progress as you can see here. As a work around, I had to select keys/certificates from the a store as part of the key generation mechanism and it is not really intended for that.
If you decide to go this route, then be prepared for a few weeks of trial and error.

Answer 2

Here is how you can by-pass openssl validation rules by providing an EC_KEY with a public key set equal to that of public cert and the private key set to any non-zero value (in my example I've just set it equal to the X coordinate of the public key). After the key is created and stored in a file, it can be passed as a regular private key to SSL_Context.
I think, idealistically openssl should address this issue in a more systematic and transparent way, but until it's done, the suggested solution can be used as a work around:
#include <string.h>
#include <stdio.h>
#include <openssl/ssl.h>
#include <openssl/x509v3.h>
static char * my_prog = "dummykey";
static char * key_file = NULL;
static char * cert_file = NULL;
int verbose = 0;
static void print_help() {
fprintf(stderr,"Version: %s\nUSAGE: %s -cert in_cert_file -key out_key_file\n",
VERSION, my_prog);
}
static void parse_args(int argc, char** argv) {
argc--;
argv++;
while (argc >= 1) {
if (!strcmp(*argv,"-key")) {
key_file = *++argv;
argc--;
}
else if (!strcmp(*argv,"-cert")) {
cert_file = *++argv;
argc--;
}
else if (!strcmp(*argv,"-v")) {
verbose = 1;
}
else {
fprintf(stderr, "%s: Invalid param: %s\n", my_prog, *argv);
print_help();
exit(1);
}
argc--;
argv++;
}
if (key_file == NULL || cert_file == NULL ) {
print_help();
exit(1);
}
}
int get_curve_nid(X509 *c) {
int ret = 0;
if (c->cert_info->key->algor->parameter) {
ASN1_TYPE *p = c->cert_info->key->algor->parameter;
if (p && p->type == V_ASN1_OBJECT) {
ret = OBJ_obj2nid(c->cert_info->key->algor->parameter->value.object);
}
}
return ret;
}
int main(int argc, char** argv) {
X509 *c=NULL;
FILE *fp=NULL;
FILE *ofp=NULL;
EC_POINT *ec_point = NULL;
BIGNUM *x = NULL;
BIGNUM *y = NULL;
EC_KEY *ec_key = NULL;
EC_GROUP *grp = NULL;
parse_args(argc, argv);
fp = fopen(cert_file, "r");
if (!fp) {
fprintf(stderr,"%s: Can't open %s\n", my_prog, cert_file);
return 1;
}
c = PEM_read_X509 (fp, NULL, (int (*) ()) 0, (void *) 0);
if (c) {
x = BN_new();
y = BN_new();
int len = c->cert_info->key->public_key->length-1;
BN_bin2bn(c->cert_info->key->public_key->data+1, len/2, x);
BN_bin2bn(c->cert_info->key->public_key->data+1+len/2, len/2, y);
EC_GROUP *grp = EC_GROUP_new_by_curve_name(get_curve_nid(c));
ec_key = EC_KEY_new();
int sgrp = EC_KEY_set_group(ec_key, grp);
int sprk = EC_KEY_set_private_key(ec_key, x);
if (sgrp && sprk) {
ec_point = EC_POINT_new(grp);
int ac = EC_POINT_set_affine_coordinates_GFp(grp, ec_point, x, y, BN_CTX_new());
int spub =EC_KEY_set_public_key(ec_key, ec_point);
ofp = fopen(key_file, "w");
int r = 0;
if (ofp) {
r = PEM_write_ECPrivateKey(ofp, ec_key, NULL, NULL, 0, NULL, NULL);
if (!r)
fprintf(stderr,"%s: Can't write EC key %p to %s\n", my_prog, ec_key, key_file);
}
else {
fprintf(stderr,"%s: Can't open %s\n", my_prog, key_file);
}
}
}
if (ec_key)
EC_KEY_free(ec_key);
if (grp)
EC_GROUP_free(grp);
if (x)
BN_free(x);
if (y)
BN_free(y);
if (c)
X509_free (c);
if (fp)
fclose(fp);
if (ofp)
fclose(ofp);
return 0;
}