I agree that CryptoAPI is nasty, but if your key doesn't have the CRYPT_EXPORTABLE flag set when it was created then it can not be exported by CryptoAPI, and can only be accessed maliciously by breaking the CryptoAPI (by patching it in memory, which requires full admin access to a running system) or accessing the OS disk offline.
CryptoAPI does not hand over the key and delegate signing and encryption to the application. Instead, it provides APIs for signing and encryption and does not disclose the private key.