fix: signing baskets follow NextGenPSD2 1.3.16 and belong to the TPP that created them

docs/signing_basket_operations.md

@@ -0,0 +1,111 @@
+# Signing baskets: operating notes
+
+For whoever runs an OBP-API instance that offers the Berlin Group signing basket. The behaviour of the API
+itself is in the ResourceDocs; this covers what an operator does and sees.
+
+## Turning authorisation on
+
+`signing_basket_authorisation_enabled` is `false` by default. While it is, creating, reading, deleting a
+basket and starting an authorisation work, and answering the authorisation (the PUT) answers 403
+`SERVICE_BLOCKED`. Roll it out in stages: first the ownership guard (this change with the property off),
+then the property, with the recovery rehearsal below in between.
+
+Related properties:
+
+| Property | Default | Meaning |
+|---|---|---|
+| `signing_basket_member_max_attempts` | 3 | Times a payment that failed to book is claimed again. Mapped connector only. |
+| `signing_basket_resume_interval_in_seconds` | 593 | How often stopped executions are resumed. 0 switches it off. |
+| `signing_basket_execution_lease_in_seconds` | 300 | How long a basket or member may sit without moving before it counts as stopped. |
+
+## What the stored status means
+
+A basket reports `RCVD`, `PATC`, `ACTC`, `CANC` or `RJCT`. Two more are stored and reported as `RCVD`:
+
+* `AUTHORISING`: the authorisation was answered correctly and the basket was claimed; its members are being
+  executed.
+* `EXECUTION_INCOMPLETE`: execution stopped with a member that is not `DONE`.
+
+`ACTC` means every member is `DONE`. Each member has its own state in `SigningBasketMemberExecution`, and the
+creating TPP reads it from `GET /signing-baskets/{basketId}/execution`.
+
+| Member state | Meaning |
+|---|---|
+| `PENDING` | Not started. |
+| `EXECUTING` | Claimed by an executor. Past the lease it becomes `UNKNOWN`. |
+| `DONE` | Payment booked, or consent activated. |
+| `FAILED` | Refused before it took effect. Claimed again automatically, on the mapped connector, up to the attempts allowed. |
+| `UNKNOWN` | The executor stopped without recording an outcome, or a connector other than the mapped one failed after it may have booked. |
+
+Several payments in one basket are not one transaction. A failure leaves the earlier payments booked.
+
+## Looking at baskets that did not complete
+
+```sql
+-- Baskets that did not reach ACTC after their authorisation was answered
+SELECT basketid, status, consumerid, psuuserid, updatedat
+FROM signingbasket
+WHERE status IN ('AUTHORISING', 'EXECUTION_INCOMPLETE')
+ORDER BY updatedat;
+
+-- What happened to each member of one basket
+SELECT membertype, memberid, position, state, detail, attempts, updatedat
+FROM SigningBasketMemberExecution
+WHERE basketid = '<basket id>'
+ORDER BY position;
+```
+
+## Members left UNKNOWN
+
+The resumption reconciles an `UNKNOWN` payment by its transaction id: if the payment carries one it was
+booked, and the member becomes `DONE`. Without one, nothing proves whether it was booked, so it is left for you.
+
+1. Find the payment's debit in the ledger (the debtor account, the amount, the time of the execution).
+2. If it was booked, set the payment's transaction id and mark the member `DONE`; the next resumption completes
+   the basket. If it was not, mark the member `FAILED`; it is then claimed again, up to the attempts allowed.
+
+On a connector other than the mapped one, a failure is always recorded as `UNKNOWN`, because that connector
+may have booked before it failed, and automatic retry is off.
+
+## Baskets created before ownership was recorded
+
+Baskets created before this change have no creating TPP and no PSU. They cannot be attributed safely, so every
+operation answers them as unknown (403 `RESOURCE_UNKNOWN`), they are never authorised, and their rows are kept.
+Nothing assigns one to whoever asks first, and no property re-opens them.
+
+To find them:
+
+```sql
+SELECT basketid, status, createdat
+FROM signingbasket
+WHERE consumerid IS NULL OR consumerid = '';
+```
+
+If a particular one has to be revived, assign it explicitly, after establishing who created it:
+
+```sql
+UPDATE signingbasket
+SET consumerid = '<consumer id of the TPP that created it>'
+WHERE basketid = '<basket id>' AND (consumerid IS NULL OR consumerid = '');
+```
+
+Its payments and consents are not held by any claim. If the basket is to be used, add the claims:
+
+```sql
+INSERT INTO SigningBasketMemberClaim (memberkey, basketid, createdat, updatedat)
+VALUES ('payment:<payment id>', '<basket id>', CURRENT_TIMESTAMP, CURRENT_TIMESTAMP);
+```
+
+(one row per payment, `consent:<consent id>` for consents). Without them another basket could take the same
+payment. A basket that is only to be closed needs none of this; set its status to `CANC`.
+
+## Things that can still surprise
+
+* A payment waiting in a basket is still a payment waiting for SCA: if
+  `berlin_group_outdated_transactions_interval_in_seconds` is set, the outdated-payment task rejects it after
+  `berlin_group_outdated_transactions_time_in_seconds`, and the basket's member then fails as not waiting for
+  authorisation.
+* A consent in a basket is `received` until activated, and the consent scheduler rejects a Berlin Group consent
+  that stays `received` for `berlin_group_outdated_consents_time_in_seconds`.
+* The recurrence of a periodic payment is not stored, so a periodic payment cannot be recognised and refused when
+  it is put in a basket; it is treated as a single payment.

obp-api/src/main/resources/props/sample.props.template

@@ -1672,9 +1672,27 @@ default_auth_context_update_request_key=CUSTOMER_NUMBER
 ## Berlin Group Create Consent Frequency per Day Upper Limit
 #berlin_group_frequency_per_day_upper_limit = 4
 
-## Berlin Group Create Consent ASPSP-SCA-Approach response header value
+## Berlin Group ASPSP-SCA-Approach response header value (create consent, create signing basket and its authorisations)
 #berlin_group_aspsp_sca_approach = redirect
 
+# Whether a Berlin Group signing basket may be authorised (PUT /signing-baskets/{basketId}/authorisations/{authorisationId}).
+# Default false: the call answers 403 SERVICE_BLOCKED. Answering the authorisation books the basket's payments
+# one after another and records, per payment, whether it was booked. Roll this out in stages: first the ownership
+# guard, then this, with a recovery rehearsal in between. Creating, reading, starting an authorisation on and
+# deleting baskets are not affected.
+#signing_basket_authorisation_enabled = false
+
+# How many times a payment of a signing basket that failed to book is claimed again, on the mapped connector only.
+# A payment on any other connector whose booking failed is left UNKNOWN for an operator.
+#signing_basket_member_max_attempts = 3
+
+# Resuming signing basket executions that stopped (for instance because the process booking the payments died).
+# The interval is in seconds (0 switches it off); the lease is how long a basket or member may sit without moving
+# before it counts as stopped. A member left executing past the lease becomes UNKNOWN and is reconciled by its
+# transaction id, or left for an operator.
+#signing_basket_resume_interval_in_seconds = 593
+#signing_basket_execution_lease_in_seconds = 300
+
 # Support multiple brands on one instance. Note this needs checking on a clustered environment
 #brands_enabled=false
 

obp-api/src/main/scala/bootstrap/liftweb/Boot.scala

@@ -125,7 +125,7 @@ import code.regulatedentities.attribute.RegulatedEntityAttribute
 import code.counterpartyattribute.{CounterpartyAttribute => CounterpartyAttributeMapper}
 import code.scheduler._
 import code.scope.{MappedScope, MappedUserScope, Scope}
-import code.signingbaskets.{MappedSigningBasket, MappedSigningBasketConsent, MappedSigningBasketPayment}
+import code.signingbaskets.{MappedSigningBasket, MappedSigningBasketConsent, MappedSigningBasketMemberClaim, MappedSigningBasketMemberExecution, MappedSigningBasketPayment}
 import code.socialmedia.MappedSocialMedia
 import code.standingorders.StandingOrder
 import code.taxresidence.MappedTaxResidence
@@ -643,6 +643,7 @@ class Boot extends MdcLoggable {
     }
     ConsentScheduler.startAll()
     TransactionScheduler.startAll()
+    SigningBasketScheduler.startAll()
 
 
     code.metrics.MetricsProps.enableMetricsScheduler match {
@@ -1004,6 +1005,8 @@ object ToSchemify extends MdcLoggable {
     MappedSigningBasket,
     MappedSigningBasketPayment,
     MappedSigningBasketConsent,
+    MappedSigningBasketMemberClaim,
+    MappedSigningBasketMemberExecution,
     MappedRegulatedEntity,
     AtmAttribute,
     AbacRule,

obp-api/src/main/scala/code/api/berlin/group/ConstantsBG.scala

@@ -47,5 +47,22 @@ object ConstantsBG {
     // 4) CANC (Cancelled) and
     // 5) RJCT (Rejected) are supported for signing baskets.
     val RCVD, PATC, ACTC, CANC, RJCT = Value
+
+    /**
+     * Stored between the moment a correct answer claims the basket and the moment its members have
+     * been dealt with. It is never reported: to a TPP a basket in this state is still RCVD, since the
+     * authorisation has not completed from its point of view.
+     */
+    val AUTHORISING_INTERNAL = "AUTHORISING"
+
+    /**
+     * Stored when the authorisation was answered correctly but not every member took effect: a payment
+     * failed, or an outcome is not known. Reported as RCVD, like AUTHORISING. The members' own results say
+     * what happened to each.
+     */
+    val EXECUTION_INCOMPLETE_INTERNAL = "EXECUTION_INCOMPLETE"
+
+    def external(storedStatus: String): String =
+      if (storedStatus == AUTHORISING_INTERNAL || storedStatus == EXECUTION_INCOMPLETE_INTERNAL) RCVD.toString else storedStatus
   }
 }

obp-api/src/main/scala/code/api/berlin/group/v1_3/BerlinGroupConsentActivation.scala

@@ -0,0 +1,62 @@
+/**
+Open Bank Project - API
+Copyright (C) 2011-2026, TESOBE GmbH.
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU Affero General Public License as published by
+the Free Software Foundation, either version 3 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU Affero General Public License for more details.
+
+You should have received a copy of the GNU Affero General Public License
+along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+Email: contact@tesobe.com
+TESOBE GmbH.
+Osloer Strasse 16/17
+Berlin 13359, Germany
+
+This product includes software developed at
+TESOBE (http://www.tesobe.com/)
+
+  */
+
+package code.api.berlin.group.v1_3
+
+import code.api.util.APIUtil.unboxFullOrFail
+import code.api.util.ErrorMessages.{ConsentAccountAccessCannotBeGranted, ConsentUpdateStatusError}
+import code.api.util.{CallContext, Consent}
+import code.consent.{ConsentStatus, ConsentTrait, Consents}
+import com.openbankproject.commons.ExecutionContext.Implicits.global
+import com.openbankproject.commons.model.User
+
+import scala.concurrent.Future
+
+/**
+ * Makes a Berlin Group consent valid once the PSU's SCA for it has succeeded: grants the access an
+ * "allAccounts" consent leaves open, marks the consent valid, and binds it to the PSU.
+ *
+ * The consent authorisation (PUT /consents/{id}/authorisations/{id}) performs the same steps inline,
+ * interleaved with checking the answer. A signing basket checks the answer once for all its members and
+ * then activates each consent here. The order is the one the consent route documents: grant before the
+ * status changes, so a consent never becomes valid without the access it names; bind last.
+ *
+ * Idempotent, so that a basket resumed after a stop can run it again: a consent already valid and bound
+ * to this PSU is returned as it is.
+ */
+object BerlinGroupConsentActivation {
+
+  def activate(consent: ConsentTrait, psu: User, callContext: Option[CallContext]): Future[ConsentTrait] =
+    if (consent.status == ConsentStatus.valid.toString && consent.userId == psu.userId) Future.successful(consent)
+    else for {
+      _ <- Consent.grantBerlinGroupAvailableAccountsAccess(psu, consent)
+        .map(unboxFullOrFail(_, callContext, ConsentAccountAccessCannotBeGranted))
+      valid <- Future(Consents.consentProvider.vend.updateConsentStatus(consent.consentId, ConsentStatus.valid))
+        .map(unboxFullOrFail(_, callContext, ConsentUpdateStatusError))
+      _ <- Consent.bindBerlinGroupConsentToPsu(consent.consentId, psu, callContext)
+    } yield valid
+}

obp-api/src/main/scala/code/api/berlin/group/v1_3/BerlinGroupPaymentAccess.scala

@@ -0,0 +1,85 @@
+/**
+Open Bank Project - API
+Copyright (C) 2011-2026, TESOBE GmbH.
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU Affero General Public License as published by
+the Free Software Foundation, either version 3 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU Affero General Public License for more details.
+
+You should have received a copy of the GNU Affero General Public License
+along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+Email: contact@tesobe.com
+TESOBE GmbH.
+Osloer Strasse 16/17
+Berlin 13359, Germany
+
+This product includes software developed at
+TESOBE (http://www.tesobe.com/)
+
+  */
+
+
+package code.api.berlin.group.v1_3
+
+import code.api.util.APIUtil.OBPReturnType
+import code.api.util.ErrorMessages.PaymentNotInitiatedByCaller
+import code.api.util.{CallContext, Consent, NewStyle}
+import code.transactionrequests.TransactionRequests
+import code.util.Helper
+import com.openbankproject.commons.ExecutionContext.Implicits.global
+import com.openbankproject.commons.model.{TransactionRequest, TransactionRequestId}
+
+/**
+ * Who may address a Berlin Group payment. Shared by the payment initiation routes and by the
+ * signing basket, which has to decide the same question about every payment it is asked to hold.
+ */
+object BerlinGroupPaymentAccess {
+
+  /**
+   * Fetch a payment the caller is entitled to address.
+   *
+   * Berlin Group names a payment by its id alone — there is no account in the path — so nothing in
+   * the route ties the payment to whoever is calling. Fetching one must therefore also establish
+   * that the caller is the party that lodged it; otherwise any authenticated TPP holding a paymentId
+   * could read another TPP's payment, list or start authorisations on it, or cancel it. Under
+   * NextGenPSD2 a payment initiation resource belongs to the TPP that created it, and only that TPP
+   * addresses it afterwards.
+   *
+   * Two things have to line up, because Berlin Group binds a payment to the TPP and the ASPSP
+   * separately knows which PSU it is for.
+   *
+   *  - The TPP. The consumer that lodged the payment is recorded on it, and a caller presenting a
+   *    different one is refused even when it is acting for the same PSU: one TPP's mandate over a
+   *    payment is not another's. Payments lodged before the consumer was recorded carry none, and
+   *    fall back to the person check alone rather than becoming unaddressable.
+   *  - The person. A payment records the principal that lodged it and, when it was lodged under a
+   *    consent, the PSU it was lodged for; a caller presents the same two. Any overlap is enough, so
+   *    a payment lodged on a client-credentials token can still be authorised under the PSU's token
+   *    and the other way round. A payment carrying neither identity belongs to nobody.
+   */
+  def getOwnPayment(paymentId: String, callContext: Option[CallContext]): OBPReturnType[TransactionRequest] =
+    for {
+      (transactionRequest, callContext) <- NewStyle.function.getTransactionRequestImpl(TransactionRequestId(paymentId), callContext)
+      initiators = Set(transactionRequest.user_id, transactionRequest.on_behalf_of_user_id).flatten.filter(_.nonEmpty)
+      callers = callContext.toSet[CallContext].flatMap(cc => cc.user.toOption.map(_.userId) ++ Consent.actingPsu(cc).map(_.userId))
+      callingConsumer = callContext.flatMap(_.consumer.map(_.consumerId.get))
+      // Read straight off the stored row rather than through the TransactionRequest model: which
+      // TPP lodged a payment is this guard's business, not something every REST connector needs on
+      // the wire, and that model's shape is a frozen contract.
+      lodgedByConsumer = TransactionRequests.transactionRequestProvider.vend
+        .getMappedTransactionRequest(TransactionRequestId(paymentId))
+        .toOption.flatMap(tr => Consent.present(tr.mConsumerId.get))
+      sameTpp = lodgedByConsumer.forall(lodgedBy => callingConsumer.contains(lodgedBy))
+      _ <- Helper.booleanToFuture(s"$PaymentNotInitiatedByCaller Payment id: $paymentId.", 403, callContext) {
+        sameTpp && initiators.exists(callers)
+      }
+    } yield (transactionRequest, callContext)
+
+}

obp-api/src/main/scala/code/api/berlin/group/v1_3/Http4sBGv13AIS.scala

@@ -533,23 +533,8 @@ object Http4sBGv13AIS extends MdcLoggable {
       }
   }
 
-  /**
-   * The PSU-ID request header, resolved to the user id it names.
-   *
-   * Berlin Group makes the header conditional rather than mandatory, so absent is a conforming
-   * answer and gives None -- the caller may be identifying the PSU some other way, which
-   * Consent.resolveBerlinGroupPsu works out. A value the ASPSP cannot resolve is a different matter
-   * and is refused with the code the standard reserves for exactly it: PSU_CREDENTIALS_INVALID, 401,
-   * "PSU-ID cannot be found by ASPSP".
-   */
   private def resolvePsuIdHeader(cc: CallContext, callContext: Option[CallContext]): Future[Option[String]] =
-    Option(APIUtil.getRequestHeader(RequestHeader.`PSU-ID`, cc.requestHeaders)).map(_.trim).filter(_.nonEmpty) match {
-      case None => Future.successful(None)
-      case Some(psuId) =>
-        Future(Consent.findPsuByPsuId(psuId)) map { psu =>
-          Some(unboxFullOrFail(psu, callContext, UserNotFoundByProviderAndUsername, 401).userId)
-        }
-    }
+    Consent.resolvePsuIdHeader(cc, callContext)
 
   // ── POST /consents/CONSENTID/authorisations (3 body-guard variants) ─────
   lazy val startConsentAuthorisationAll: HttpRoutes[IO] = HttpRoutes.of[IO] {
@@ -719,16 +704,7 @@ object Http4sBGv13AIS extends MdcLoggable {
             _ <- NewStyle.function.tryons(ConsentUpdateStatusError, 400, callContext) {
               consent.toList.size == 1
             }
-            _ <- Future {
-              val authContexts = UserAuthContextProvider.userAuthContextProvider.vend.getUserAuthContextsBox(psu.userId)
-                .map(_.map(i => BasicUserAuthContext(i.key, i.value)))
-              ConsentAuthContextProvider.consentAuthContextProvider.vend.createOrUpdateConsentAuthContexts(consentId, authContexts.getOrElse(Nil))
-            } map {
-              unboxFullOrFail(_, callContext, ConsentUserAuthContextCannotBeAdded)
-            }
-            _ <- Future(Consents.consentProvider.vend.updateConsentUser(consentId, psu)) map {
-              unboxFullOrFail(_, callContext, ConsentUserCannotBeAdded)
-            }
+            _ <- Consent.bindBerlinGroupConsentToPsu(consentId, psu, callContext)
           } yield {
             createPutConsentResponseJson(consent.toList.head)
           }