How to specify Mask Generation Function in RSAES-OAEP using CNG APIs? - cryptography

Question

I’m implementing RSA-OAEP with windows CNG libraries. So far I’ve been able to have a full (encryption/decryption) flow with CNG library and able to verify results with OpenSSL. However, this only works if the hashing function is the same as the MGF1. If these two are different my implementation with CNG fails, For example, if OpenSSL command is changed from:
pkeyutl -encrypt -in test.txt -pubin -inkey keypair.pem -out out.bin -pkeyopt rsa_padding_mode:oaep -pkeyopt rsa_oaep_md:sha256 -pkeyopt rsa_mgf1_md:sha256
To:
pkeyutl -encrypt -in test.txt -pubin -inkey keypair.pem -out out.bin -pkeyopt rsa_padding_mode:oaep -pkeyopt rsa_oaep_md:sha256 -pkeyopt rsa_mgf1_md:sha1
CNG will fail to decrypt (Note the change from SHA256 to SHA1 in mgf1 parameter). My guess is that I need to specify the use of SHA1 as the mask generation function to CNG APIs, but I'm unable to figure out how to do so. So far my research has pointed the existence of the CRYPT_RSAES_OAEP_PARAMETERS struct which allows to specify the mask generation function. But I haven’t been able to find a sample on how to use this parameters with CNG.
Any help, deeply appreciated.
Here is my CNG code:
BCRYPT_OAEP_PADDING_INFO paddingParams = { BCRYPT_SHA256_ALGORITHM, NULL, 0 };
///Encryption
status = BCryptEncrypt(hKey, pbInput, cbInput, &paddingParams, NULL /*pbIV*/, 0 /*cbIV*/, NULL /*pbOutput*/, 0 /*cbOutput*/, &cbBuffer, BCRYPT_PAD_OAEP);
if (!NT_SUCCESS(status))
{
printf("Failed to get required size of buffer..status : %08x\n", status);
}
pbBuffer = (PUCHAR) LocalAlloc(0, cbBuffer);
status = BCryptEncrypt(hKey, pbInput, cbInput, &paddingParams, NULL /*pbIV*/, 0 /*cbIV*/, pbBuffer, cbBuffer, &cbBuffer, BCRYPT_PAD_OAEP);
if (!NT_SUCCESS(status))
{
printf("Failed encrypt data..status : %08x\n", status);
}
//Decryption
status = BCryptDecrypt(hKey, pbBuffer, cbBuffer, &paddingParams, NULL/*pbIV*/, 0/*cbIV*/, NULL, 0, &cbBufferRaw, BCRYPT_PAD_OAEP);
if (!NT_SUCCESS(status))
{
printf("Failed to get required size of buffer..status : %08x\n", status);
}
pBufferRaw = (PUCHAR) LocalAlloc(0, cbBufferRaw);
status = BCryptDecrypt(hKey, pbBuffer, cbBuffer, &paddingParams, NULL/*pbIV*/, 0/*cbIV*/, pBufferRaw, cbBufferRaw, &cbBufferRaw, BCRYPT_PAD_OAEP);
if (!NT_SUCCESS(status))
{
printf("Failed to get required size of buffer..status : %08x\n", status);
}