The start-time key wrapping check wraps with a 1024-bit RSA key, which BC-FIPS refuses in approved-only mode

[vharseko]

Problem

The crypto manager checks its key wrapping transformation when it is created (at every server start) and on every change of key-wrapping-transformation. The check (CryptoManagerImpl.isConfigurationChangeAcceptable) wraps a freshly generated MAC key with the public key of a hard-coded dummy certificate, since the trust store backend is not available yet. That certificate carries a 1024-bit RSA key (and is signed with MD5withRSA).

OpenDJ registers the Bouncy Castle FIPS provider at position 1 (FipsStaticUtils), so that provider answers the cipher request. In approved-only mode (-Dorg.bouncycastle.fips.approved_only=true) the wrap then fails. RSA keys under 2048 bits are not approved for key transport:

org.bouncycastle.crypto.fips.FipsUnapprovedOperationError: Attempt to use RSA key size outside of accepted range - requested keySize 1024 bits: RSA/OAEP
	at org.bouncycastle.crypto.fips.FipsRSA.validateKeySize(Unknown Source)
	...
	at javax.crypto.Cipher.init(Cipher.java:1296)

(bc-fips 2.1.3, the version the build ships, on JDK 26; RSA/ECB/OAEPWITHSHA-1ANDMGF1PADDING from the BCFIPS provider, key from SunRsaSign. The same wrap with a 2048-bit key succeeds, and so does the 1024-bit wrap without approved-only mode.)

FipsUnapprovedOperationError is an Error, and the check catches Exception. So it is not reported as ERR_CRYPTOMGR_CANNOT_GET_PREFERRED_KEY_WRAPPING_CIPHER: it escapes from the crypto manager's initialization. The transformation itself is fine, so no value of key-wrapping-transformation gets such a runtime past the check.

The same check also computes the instance key identifier with MessageDigest MD5 (getInstanceKeyID). That works as long as a provider still offers MD5, but it is one more thing a restricted runtime can refuse.

Expected

The validation wrap uses a key which an approved-only provider accepts (at least 2048 bits), and a provider refusing the operation is reported as a configuration problem rather than escaping as an Error.

Notes

• OpenDJ does not turn approved-only mode on itself. The install guide (Keep PKCS5S2 usable on a FIPS-restricted JCE, and name the key wrapping property when the runtime has no RSA-OAEP #1058) says the default configuration works with the bundled BC-FIPS provider in its default mode.
• Found in the review of Keep PKCS5S2 usable on a FIPS-restricted JCE, and name the key wrapping property when the runtime has no RSA-OAEP #1058, which splits this check so that only a refused transformation names the key-wrapping-transformation property. Related: Crypto manager: a FIPS-approved key transport for JVMs whose only RSA cipher is PKCS#1 v1.5 (SunPKCS11) #1056.