Intermittent SCR 'Could not obtain lock' in api-servlet during startup fails ui-smoke-tests

[vharseko]

Symptom

The ui-smoke-tests (17, /myidm, samples/usecase/usecase1) job failed on PR #227 (run 35873142464, job 107389094758). The UI tests themselves passed. The job was failed by the step that greps openidm/logs/openidm0.log.0 for ERROR|SEVERE|Exception|Throwable. It found one SCR error logged during startup:

Sep 23, 2026 11:43:47 PM org.forgerock.openidm.logging.LogServiceTracker logEntry
SEVERE: Bundle: org.openidentityplatform.openidm.api-servlet [187] FrameworkEvent ERROR
org.apache.felix.log.LogException: org.osgi.framework.ServiceException: Service factory exception: Could not obtain lock
	at org.apache.felix.framework.ServiceRegistrationImpl.getFactoryUnchecked(ServiceRegistrationImpl.java:354)
	...
	at org.apache.felix.scr.impl.manager.SingleRefPair.getServiceObject(SingleRefPair.java:88)
	at org.apache.felix.scr.impl.manager.DependencyManager$SingleStaticCustomizer.prebind(DependencyManager.java:1207)
	at org.apache.felix.scr.impl.manager.AbstractComponentManager.collectDependencies(AbstractComponentManager.java:1030)
	at org.apache.felix.scr.impl.manager.SingleComponentManager.getServiceInternal(SingleComponentManager.java:937)
	at org.apache.felix.scr.impl.manager.AbstractComponentManager.activateInternal(AbstractComponentManager.java:757)
	at org.apache.felix.scr.impl.manager.DependencyManager$SingleStaticCustomizer.addedService(DependencyManager.java:1091)
	...
	at org.apache.felix.framework.BundleContextImpl.registerService(BundleContextImpl.java:328)
	at org.apache.felix.scr.impl.manager.AbstractComponentManager.registerService(AbstractComponentManager.java:960)
	at org.apache.felix.scr.impl.manager.AbstractComponentManager.activateInternal(AbstractComponentManager.java:733)
	at org.apache.felix.scr.impl.manager.DependencyManager$SingleStaticCustomizer.addedService(DependencyManager.java:1091)
	...
	at org.apache.felix.framework.BundleContextImpl.registerService(BundleContextImpl.java:328)
	at org.ops4j.pax.web.service.internal.Activator.performConfiguration(Activator.java:597)
	at org.ops4j.pax.web.service.internal.Activator.updateController(Activator.java:463)
	at org.ops4j.pax.web.service.internal.Activator.lambda$updateServerControllerFactory$1(Activator.java:365)
	...
Caused by: java.lang.IllegalStateException: Could not obtain lock
	at org.apache.felix.scr.impl.manager.AbstractComponentManager.obtainLock(AbstractComponentManager.java:241)
	at org.apache.felix.scr.impl.manager.AbstractComponentManager.obtainStateLock(AbstractComponentManager.java:288)
	at org.apache.felix.scr.impl.manager.SingleComponentManager.getService(SingleComponentManager.java:890)
	at org.apache.felix.framework.ServiceRegistrationImpl.getFactoryUnchecked(ServiceRegistrationImpl.java:349)

PR #227 only changes openidm-shell (UpdateCommand and related classes) and the docs, so it does not touch this code path. The other 29 checks of the same run passed, including the same sample on JDK 26 and the same sample without /myidm. The failure is intermittent.

What the trace shows

• The pax-web configuration thread (Activator.performConfiguration) registers the HTTP service. SCR then synchronously activates a chain of components with static references that depend on it (two nested SingleStaticCustomizer.addedService → activateInternal frames). The error is attributed to the api-servlet bundle.
• While collecting the static dependencies of a component in that chain (collectDependencies → prebind), SCR calls getService on a referenced component. That component's state lock is held by another thread. SCR gives up after its lock timeout (5 s by default) with Could not obtain lock. The timestamps match: Jetty reports Started oejs.Server at 23:43:42, the error is logged at 23:43:47.
• Right before that, the log shows the activation of AuthenticationService (service.pid=org.forgerock.openidm.authentication, component.id=37). AuthenticationService configures AuthFilterWrapper (service.pid=org.forgerock.openidm.auth.config). ServletComponent holds a static reference to that same filter (openidm-api-servlet/.../ServletComponent.java:113). This makes AuthFilterWrapper / AuthenticationService the likely holder of the contended lock, but the log does not name the component, so this is not confirmed.

Related

• Fix SCR deadlock in SecurityManager by making repoService a dynamic reference #169 fixed a similar SCR deadlock in SecurityManager by making repoService a dynamic reference.
• [#225] Bind identity provider configs through the DS bind methods #226 changes how AuthenticationService binds IdentityProviderService (dynamic reference, rebuilds the auth modules on bind). It is in the same area but does not address this lock.

Next steps

1. Identify the component whose state lock times out. Options: enable SCR debug logging (ds.loglevel=debug) in the smoke-test job, or take a thread dump when the error appears.
2. Break the chain of synchronous static activations that runs on the pax-web thread. For example, make ServletComponent.authFilter a dynamic reference, the same way Fix SCR deadlock in SecurityManager by making repoService a dynamic reference #169 fixed SecurityManager, so that ServletComponent activation no longer waits for the auth filter's lock.
3. Until this is fixed, the log check in ui-smoke-tests will keep failing intermittently on this error. The error must not be added to the allow-list: that would hide a real startup race.