How to decrypt data using RSACng that is previously encrypted with RSACryptoServiceProvider - cryptography

Question

I am migrating on RSACng for new version release from RSACryptoServiceProvider. However, as RSACryptoserviceProvider that is CAPI uses Little Endian Architecture and RSACng that is CNG API is using Big Endian Architecture, question is how can i decrypt the data using CNG Api that is previously encrypted using RSACryptoService provider (CAPI)?
I have already tried Array.reverse(cypherText) and tried to decrypt using CNG Api, but it is throwing the error, 'The parameter is incorrect'.
I have also tried the approach of decrypting half the cypher text because CNG API is using RSAEncryptionPadding.OaepSHA512 padding whereas CAPI uses OAEP padding.
My RSACryptoServiceProvider class is as below:-
public static void EncryptWithSystemKeyRSACryptoService(byte[]
plainBytes, bool representsUnicodeString, out string cypherText)
{
CspParameters cp = new CspParameters();
cp.KeyContainerName = regValue.ToString();
cp.Flags = CspProviderFlags.UseMachineKeyStore;
cp.KeyNumber = (int)KeyNumber.Exchange;
byte[] encBlockData=null;
using (RSACryptoServiceProvider rsaCSP = new RSACryptoServiceProvider(cp))
{
res = CryptResult.GeneralError;
int keysize = rsaCSP.KeySize;
//This encrypts data and uses FOAEP padding
encBlockData = rsaCSP.Encrypt(plainBytes, true);
}
//Should i have to reverse the Byte order?
// I am doing Array.reverse for encrypted data as it follows little endian architecture and CNG Api follows Big Endian architecture
Array.Reverse(encBlockData);
cypherText = BitConverter.ToString(encBlockData );
cypherText = cypherText.Replace("-", "");
cypherText = cypherText.ToLower();
}
This is how i encrypt data with RSACryptoservice Provider (CAPI)
My RSACng class is as below :-
//I am calling this to use RSACng API to get private keys
private static byte[] SetPrivateAndPublicKeysAndDecrypt(byte[] cyphertext)
{
cp.KeyContainerName = regValue.ToString();
cp.Flags = CspProviderFlags.UseMachineKeyStore;
cp.KeyNumber = (int)KeyNumber.Exchange;
using (RSACryptoServiceProvider rsaCSP = new
RSACryptoServiceProvider(cp))
{
res = CryptResult.GeneralError;
keysize = rsaCSP.KeySize;
q = rsaCSP.ExportCspBlob(false);
RSAp = rsaCSP.ExportParameters(true);
}
//created cngKey
cngKey = CngKey.Import(q, CngKeyBlobFormat.GenericPublicBlob);
//created RSACng instance
RSACng rsacng = new RSACng(cngKey)
{
KeySize = keysize
};
rsacng.ImportParameters(RSAp);
//Decrypt using RSACng API using OAEPSHA512 padding
var plainText= crypto.Decrypt(cyphertext, RSAEncryptionPadding.OaepSHA512);
return plainText;
}
Expected result should be-> plainText successfully decrypted
Actual Resul-> Exception caught-> 'The parameter is incorrect'.

Answer 1

RSA ciphertext is defined to use statically sized, unsigned, big endian encoding in PKCS#1 (which specifies PKCS#1 v1.5 RSA encryption and OAEP encryption as implemented by most libraries). The function is called I2OSP within that standard, and the ciphertext should have the same size (in full bytes) as the key size. If it isn't big endian, then it does not conform to RSA / OAEP, in other words.
The same goes for normal ASN.1 encoded keys: they use dynamically sized, signed, big endian encoding according to DER (distinguished encoding rules). Those keys are defined in PKCS#1, PKCS#8 and X.509 standards, although they may also be embedded in a PKCS#12 compatible key store - for instance. Sometimes the keys are then PEM encoded as well to make them compatible with protocols that require text rather than binary.
So you should never have to reverse ciphertext or keys that use one of the standard encodings. That the calculations are performed internally on little endian (or not) is of no concern. This is true for about every modern cipher or other cryptographic primitive; the input / output is simply defined in bytes with a specific order, not numbers. Only very low level functions may possibly operate on e.g. words, which muddles the problem (but you won't find that in the MS API's).
Only Microsofts own proprietary (lame) key encodings may use little endian.
bartonjs is of course correct in the comments; you need to match the padding methods, and the default hash to be used for OAEP (or rather, the mask generation function MGF1 within OAEP) is SHA-1. There are plenty of other pitfalls to avoid, such as performing correct encoding / decoding of plaintext / ciphertext.