[#1081] Wrap with a 2048-bit key in the key wrapping check, and report a provider refusing with an Error

[vharseko]

Fixes #1081

Problem

The crypto manager checks its key wrapping transformation at every start (and on every change of key-wrapping-transformation) by wrapping a freshly generated MAC key with the public key of a hard-coded certificate. That certificate carries a 1024-bit RSA key. The BC-FIPS provider in approved-only mode refuses RSA keys under 2048 bits with FipsUnapprovedOperationError, an Error, which escaped catch (Exception) and kept the server from starting.

Change

CryptoManagerImpl.isConfigurationChangeAcceptable:

• The check wraps with a new certificate: 2048-bit RSA key, signed with SHA256withRSA, valid until 2126. Its private key was thrown away when it was generated.
• The key identifier of that wrap is a constant instead of the MD5 digest of the certificate (getInstanceKeyID). The wrapped key is thrown away, so the identifier only has to be a string, and a restricted runtime may not offer MD5.
• The cipher, MAC and key wrapping checks catch Exception | Error, so a provider refusing with an Error is reported as a refusal of the configuration (ERR_CRYPTOMGR_CANNOT_GET_REQUESTED_ENCRYPTION_CIPHER, ..._MAC_ENGINE, ..._PREFERRED_KEY_WRAPPING_CIPHER). VirtualMachineError and LinkageError are rethrown, as they were before: neither is a refusal, and a LinkageError from a broken provider jar (ExceptionInInitializerError, NoClassDefFoundError) reported as a refusal would lose its cause, since a refusal carries only a message and the start then fails with an InitializationException without a cause. BC-FIPS refuses with subclasses of AssertionError, which stay refusals. BC-FIPS in approved-only mode also refuses to generate keys with a random generator it has not approved, with an Error as well, so the MAC and cipher checks get the same treatment as the wrap.
• The javadoc of isKeyWrappingTransformationSupported (Keep PKCS5S2 usable on a FIPS-restricted JCE, and name the key wrapping property when the runtime has no RSA-OAEP #1058) no longer mentions MD5 and the 1024-bit key.

Tests

CryptoManagerTestCase. The approved-only cases run the check in a thread of its own, set to approved-only mode with CryptoServicesRegistrar.setApprovedOnlyMode(true) (per-thread, so no other test is affected), with BC-FIPS installed first for the duration:

• testKeyWrappingCheckPassesUnderApprovedOnlyBcFips: RSA-OAEP passes the check.
• testKeyWrappingRefusedWithAnErrorIsReported: RSA/ECB/PKCS1Padding, which BC-FIPS refuses in that mode with an Error, is reported with ERR_CRYPTOMGR_CANNOT_GET_PREFERRED_KEY_WRAPPING_CIPHER.
• testMacKeyGenerationRefusedWithAnErrorIsReported, testCipherKeyGenerationRefusedWithAnErrorIsReported: an Error from key generation is reported as a refusal of the MAC algorithm or the cipher transformation.
• testErrorWhichIsNotARefusalPropagates (review round 1): a probe provider fails the cipher, MAC and key wrapping checks with a StackOverflowError and with an ExceptionInInitializerError; each propagates as it is, and no refusal is recorded. For the key wrapping check the probe cipher is found and fails once initialized, so the failure comes from the wrap and not from isKeyWrappingTransformationSupported.
• testKeyWrappingCheckDoesNotNeedMd5 replaces testKeyWrappingRefusalForAnotherCauseDoesNotNameTheProperty from Keep PKCS5S2 usable on a FIPS-restricted JCE, and name the key wrapping property when the runtime has no RSA-OAEP #1058, whose refusal came from MD5: without MD5 the check now passes.

The crypto manager generates keys with a SecureRandom taken from the provider which came first when the class was loaded. In the test JVM that is SUN, and BC-FIPS in approved-only mode refuses to generate keys with it. The two key wrapping cases therefore set mac-algorithm to HmacMD5, which approved-only BC-FIPS does not offer, so SunJCE makes the MAC key and BC-FIPS still does the wrap. The MAC and cipher cases use that refusal as their Error.

Before the fix all five new cases failed: the wrap cases with the 1024-bit FipsUnapprovedOperationError, the MAC and cipher cases with the escaped Error, the MD5 case with MD5 MessageDigest not available. With the fix CryptoManagerTestCase passes 42/42; GetSymmetricKeyExtendedOperationTestCase and ConfigureDSTestCase pass too. Three mutants were each caught by their test: the MD5 key identifier, the 1024-bit certificate, and catch (Exception) in the key wrapping check. Five more were caught by testErrorWhichIsNotARefusalPropagates: removing the rethrow from the cipher, MAC or key wrapping check (2 cases each), emptying the rethrow (6), and rethrowing only VirtualMachineError (the 3 ExceptionInInitializerError cases).

Not changed

A provider refusing the wrap itself (like PKCS#1 v1.5 above) is still reported with the generic ERR_CRYPTOMGR_CANNOT_GET_PREFERRED_KEY_WRAPPING_CIPHER, which does not name the key-wrapping-transformation property. Only a transformation the runtime does not provide at all names it (#1058).

[maximthomas]

praise: The change sits where #1081 fails and removes both causes the check had.

• CryptoManagerImpl.isConfigurationChangeAcceptable (CryptoManagerImpl.java:409-431) wraps with a 2048-bit RSA key (SPKI header MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA, SHA256withRSA) and the constant key id key-wrapping-check. The check no longer needs a 1024-bit wrap or MD5.
• inApprovedOnlyBcFipsThread (CryptoManagerTestCase.java:464-474) sets approved-only mode per thread with CryptoServicesRegistrar.setApprovedOnlyMode(true), so the in-core server never runs in approved-only mode.

---

question (non-blocking): Should a LinkageError from a broken provider jar become a configuration refusal, now that the refusal drops its cause?

opendj-server-legacy/src/main/java/org/opends/server/crypto/CryptoManagerImpl.java:354, :377, :433, :451-457, :297

The widened catches take every Error except VirtualMachineError. That includes ExceptionInInitializerError and NoClassDefFoundError. Take a provider ahead of SunJCE whose KeyGeneratorSpi has a failing static initializer. It throws an ExceptionInInitializerError with a null message. getExceptionMessage does not unwrap that error. It prints only the class name and the first frame, so the reason reads ExceptionInInitializerError(<File>:<line>). Line 297 then throws new InitializationException(why.get(0)) with no cause. The stack trace goes to logger.traceException only, and the shipped configuration disables the debug logger. At the base, the same Error escaped uncaught, and the JVM printed the full stack with Caused by. The start fails in both cases, so only the diagnosis is lost. If this is not intended, it is a Minor, and the fix below restores the base behaviour for a broken jar. The BC-FIPS refusals are still reported, because FipsUnapprovedOperationError extends AssertionError.

  private static void rethrowIfVirtualMachineError(final Throwable t)
  {
    if (t instanceof VirtualMachineError || t instanceof LinkageError)
    {
      throw (Error) t;
    }
  }

---

suggestion (non-blocking): No test pins rethrowIfVirtualMachineError.

opendj-server-legacy/src/main/java/org/opends/server/crypto/CryptoManagerImpl.java:355, :378, :434, :451-457

Every new case raises BC-FIPS's FipsUnapprovedOperationError, which the helper lets through. No case in CryptoManagerTestCase makes a provider throw a VirtualMachineError. Deleting any of the three calls, or emptying the helper's body, therefore leaves all 36 cases green. With that mutant, an OutOfMemoryError or StackOverflowError from a provider becomes an ERR_CRYPTOMGR_* refusal instead of propagating.

  /** A VirtualMachineError from a provider is not a refusal of the configuration: it propagates. */
  @Test
  public void testVirtualMachineErrorFromAProviderPropagates() throws Exception
  {
    final CryptoManagerImpl cm = DirectoryServer.getCryptoManager();
    final CryptoManagerCfg cfg = getServerContext().getRootConfig().getCryptoManager();
    final Provider vme = new Provider("VmeProbe", "1.0", "KeyGenerator failing like the JVM") {};
    vme.put("KeyGenerator.HmacVmeProbe", StackOverflowingKeyGenerator.class.getName());
    Security.insertProviderAt(vme, 1);
    try
    {
      assertThatThrownBy(() -> cm.isConfigurationChangeAcceptable(
          withProperty(cfg, "getMacAlgorithm", "HmacVmeProbe"), new ArrayList<LocalizableMessage>()))
          .isInstanceOf(StackOverflowError.class);
    }
    finally
    {
      Security.removeProvider("VmeProbe");
    }
  }

  /** A key generator which fails the way only the JVM fails. */
  public static final class StackOverflowingKeyGenerator extends javax.crypto.KeyGeneratorSpi
  {
    @Override
    protected void engineInit(java.security.SecureRandom random) {}

    @Override
    protected void engineInit(java.security.spec.AlgorithmParameterSpec params, java.security.SecureRandom random) {}

    @Override
    protected void engineInit(int keySize, java.security.SecureRandom random) {}

    @Override
    protected javax.crypto.SecretKey engineGenerateKey()
    {
      throw new StackOverflowError("provider probe");
    }
  }

Pin: this case fails if the call at :378 is deleted or the helper's body is emptied. A cipher-arm twin, with a KeyGenerator for the transformation's algorithm, catches the deletion at :355. OpenJDK does not require a signed JCE provider.

[vharseko]

Both points taken; round 2 is cf821bd, rebased on the current master (d30ff78, range-diff = for the round-1 commit).

question (LinkageError): not intended. The helper, renamed rethrowIfNotARefusal, now rethrows LinkageError as well as VirtualMachineError, so an ExceptionInInitializerError or NoClassDefFoundError from a broken provider jar propagates with its cause, as it did at the base. The BC-FIPS refusals stay refusals: FipsUnapprovedOperationError, FipsSelfTestFailedError and general.OperationError all extend AssertionError (checked against bc-fips 2.1.2).

suggestion (no pin): testErrorWhichIsNotARefusalPropagates, over all three checks, each with a StackOverflowError and an ExceptionInInitializerError (6 cases). The probe provider serves a KeyGenerator for the cipher and MAC checks, and a Cipher for the key wrapping check. That cipher is found by Cipher.getInstance and fails only in engineInit, so the failure comes from the wrap at :434, not from isKeyWrappingTransformationSupported, which lets an Error through anyway. Each case also asserts that no refusal was recorded.

Mutants, CryptoManagerTestCase 42 cases:

mutant	failed
no rethrow in the cipher check (:355)	2 (cipher × SOE, EIIE)
no rethrow in the MAC check (:378)	2 (MAC × SOE, EIIE)
no rethrow in the key wrapping check (:434)	2 (wrap × SOE, EIIE)
helper body emptied	6
helper rethrows VirtualMachineError only	3 (the EIIE cases)

Without mutants, 42/42 pass. The PR description is updated.