openssl 1.0.2j, how to force server to choose ECDH* ciphers - ssl

Question

I have client server which uses opensl 1.0.2j, and using RSA:4096 key and certificate and want to force the server to use only the following ciphers.
ECDHE-RSA-AES256-GCM-SHA384
ECDHE-RSA-AES256-SHA384
ECDH-RSA-AES128-GCM-SHA256
ECDH-RSA-AES128-SHA256
DHE-RSA-AES256-GCM-SHA384
DHE-RSA-AES256-SHA256
DHE-RSA-AES128-GCM-SHA256
DHE-RSA-AES128-SHA256
My server side code will look like below.
method = SSLv23_server_method();
ctx = SSL_CTX_new(method);
SSL_CTX_set_cipher_list(ctx, "ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384:ECDH-RSA-AES128-GCM-SHA256:ECDH-RSA-AES128-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-SHA256:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES128-SHA256");
SSL_CTX_set_ecdh_auto(ctx, 1);
SSL_CTX_set_tmp_dh(ctx, dh);
SSL_CTX_set_tmp_ecdh(ctx, ecdh);
SSL_CTX_use_certificate_file(ctx, certFilePath, SSL_FILETYPE_PEM);
SSL_CTX_use_PrivateKey_file(ctx, privKeyPath, SSL_FILETYPE_PEM)
SSL_accept()
My client side code will look like as below
method = SSLv23_server_method();
ctx = SSL_CTX_new(method);
SSL_CTX_set_cipher_list(ctx, "ECDH-RSA-AES128-GCM-SHA256:ECDH-RSA-AES128-SHA256:ECDH-ECDSA-AES128-GCM-SHA256:ECDH-ECDSA-AES128-SHA256");
SSL_CTX_set_ecdh_auto(ctx, 1);
SSL_CTX_set_tmp_dh(ctx, dh);
SSL_CTX_set_tmp_ecdh(ctx, ecdh);
SSL_CTX_use_certificate_file(ctx, certFilePath, SSL_FILETYPE_PEM);
SSL_CTX_use_PrivateKey_file(ctx, privKeyPath, SSL_FILETYPE_PEM)
SSL_connect()
The last step ssl_accept() on the server fails with
err: 336027900 'error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol'
If i use ECDHE*RSA* or DHE*RSA* ciphers on the client side, it is working fine.
Could you please let me know what I am missing?
Edit: server's certificate(certFilePath) contains an RSA public key not the ECDH public key.

Answer 1

Meta: answer only up to TLS1.2; 1.3 no longer has keyexchange and authentication in ciphersuite.
First, it makes no sense to call both set_ecdh_auto and set_tmp_ecdh -- those are mutually exclusive. Also your server doesn't request client authentication, so configuring a self-cert&key on the client is useless. OTOH your server is using a selfsigned cert which probably isn't in the client's default truststore, so you may need to configure the client truststore (there are several approaches to doing that).
Second, 'static' ECDH ciphersuites are quite different from ECDHE suites, just as 'static' DH suites are different from DHE suites. Both static forms are not widely implemented and very little used because they generally offer no benefit; in particular OpenSSL 1.1.0 and up no longer implement them, so your code becomes obsolete in about a year if I remember the schedule correctly.
To be exact, DH-RSA suites require a cert containing a DH key (which permits keyAgreement), and for TLS<=1.1 the cert must be issued by a CA using an RSA key; for 1.2 this latter restriction is removed. No public CA will issue a cert for a DH key, and even doing it yourself isn't easy; see https://security.stackexchange.com/questions/44251/openssl-generate-different-type-of-self-signed-certificate and (my) https://crypto.stackexchange.com/questions/19452/static-dh-static-ecdh-certificate-using-openssl/ .
ECDH-RSA suites similarly require a cert containing an ECC key which permits keyAgreement, and issued by RSA if <=1.1; this is somewhat easier because the key (and CSR) for ECDH is the same as for ECDSA and only KeyUsage needs to differ. For your self-created and self-signed case, it's easy, just generate an ECC key and cert (automatically signed with ECDSA).
But last, this shouldn't cause 'unknown protocol'; it would cause 'no shared cipher' and handshake_failure. The code you've shown shouldn't cause 'unknown protocol', so you probably need to investigate and fix that first. You might try using commandline s_client instead especially with its -debug or -msg hooks, or -trace if you have compiled that in.