I have difficulty to decrypt data being encrypted using OpenSSL, RSA and RSA_PKCS1_OAEP_PADDING padding option.
What I am doing is to load the key from Windows KSP:
m_hSystemStore = CertOpenStore(CERT_STORE_PROV_SYSTEM, 0, NULL, CERT_SYSTEM_STORE_LOCAL_MACHINE, m_storeName.c_str());
m_pCertWithKeys = CertFindCertificateInStore(m_hSystemStore, SupportedEncodings, 0, CERT_FIND_SUBJECT_STR, m_certName.c_str(), NULL);
DWORD m_KeyContextSpec = 0;
HCRYPTPROV_OR_NCRYPT_KEY_HANDLE m_hKeyContextFull;
CryptAcquireCertificatePrivateKey(m_pCertWithKeys, CRYPT_ACQUIRE_SILENT_FLAG | CRYPT_ACQUIRE_PREFER_NCRYPT_KEY_FLAG, NULL, &m_hKeyContextFull, &m_KeyContextSpec, &m_KeyContextMustBeReleased);
and call NCryptDecrypt like:
BCRYPT_OAEP_PADDING_INFO paddingInfo = { 0 };
DWORD cbDecryptedMessage;
BYTE* pbDecryptedMessage = NULL;
paddingInfo.pszAlgId = BCRYPT_SHA1_ALGORITHM;
NCryptDecrypt(m_hKeyContextFull, (LPBYTE)pEncrypted, encryptedLenInBytes, &paddingInfo, NULL, cbDecryptedMessage, &outputDataLen, NCRYPT_PAD_OAEP_FLAG | NCRYPT_SILENT_FLAG);
NCryptDecrypt(m_hKeyContextFull, (LPBYTE)pEncrypted, encryptedLenInBytes, &paddingInfo, pbDecryptedMessage, cbDecryptedMessage, &outputDataLen, NCRYPT_PAD_OAEP_FLAG | NCRYPT_SILENT_FLAG);
It fails with NTE_INVALID_PARAMETER (0x80090027). I tried different flags but none of them works.
Note: All error checkings have been removed from code for readability.
The data is being encrypted with same key (public part) like:
RsaPublicEncrypt(size - 42, blk, output, Rsa, RSA_PKCS1_OAEP_PADDING)
and can be decrypted using SoftHSM successfully.
Is there any limitation for CNG to decrypt data being encrypted by OpenSSL?
Is there any idea what am I doing wrong?
Thanks.
What I have tried:
I did not change anything in OpenSSL side, because that part works and we can decrypt data using SoftHSM with same key, but:
* I have tried different flags in NCryptDecrypt
* Different algorithm for padding
* Different algorithm to generate the key
None of them worked so far.