Bulk deletion of orders not possible via GUI

[peterfriedl]

Use Case: I want to delete all orders for the time span 2026-04-01 to 2026-06-01 (prosEO 2.2.0 Inpuls deployment):

Procedure:
In GUI select Start Time between 2026-04-01 and 2026-06-01 and State CLOSED-> Adjust Elements per page according to "Number of elements: ***" ->Update number of elements -> Select all orders -> Handle Selected Orders -> Delete

Result after some time of processing (turning wheel):

But no orders were deleted.

[tangobravo62]

Cannot be reproduced easily in the development environment (PTM mission, just a handful of processing orders). Log analysis on original system required.

@peterfriedl : Is 2026-09-03 around 14:30 LT the correct time to check for log entries?

[peterfriedl]

@tangobravo62 I created the issue more or less at the same time of executing the described procedure. Since the issue has been created at 3 Sept. 14:48 GMT+2, I would say the answer to your question is yes. :-)

[emelchinger]

The analysis of the order and GUI log files shows no warnings or errors regarding the delete of orders. Also the database log doesn't show anything.
The order logs e.g.

2026-09-03T12:34:34.471Z TRACE 1 --- [http-nio-8080-exec-1] d.d.p.ordermgr.rest.OrderControllerImpl : >>> deleteOrderById(2758781) 2026-09-03T12:34:34.472Z TRACE 1 --- [http-nio-8080-exec-1] d.d.p.ordermgr.rest.ProcessingOrderMgr : >>> deleteOrderById(2758781) 2026-09-03T12:34:34.474Z TRACE 1 --- [http-nio-8080-exec-1] d.d.p.ordermgr.rest.ProcessingOrderMgr : >>> deleteOrder(repro_UMAS_05_29_2026_01) 2026-09-03T12:34:34.474Z TRACE 1 --- [http-nio-8080-exec-1] d.d.p.ordermgr.rest.ProcessingOrderMgr : >>> prepareOrderToDelete(repro_UMAS_05_29_2026_01) 2026-09-03T12:34:34.477Z TRACE 1 --- [http-nio-8080-exec-9] d.d.p.ordermgr.rest.ProcessingOrderMgr : >>> deleteJOF(2758833) 2026-09-03T12:34:34.514Z TRACE 1 --- [http-nio-8080-exec-3] d.d.p.ordermgr.rest.ProcessingOrderMgr : >>> deleteJOF(2770863) 2026-09-03T12:34:34.521Z INFO 1 --- [http-nio-8080-exec-18] d.d.p.ordermgr.rest.ProcessingOrderMgr : (I3525) Job Order File s3://proseo-data-inpuls/joborders/2026/6/9/15/ed2232d3-b47a-4c31-8406-49cab3c8e84a.xml deleted from processing facility inpuls 2026-09-03T12:34:34.528Z INFO 1 --- [http-nio-8080-exec-9] d.d.p.ordermgr.rest.ProcessingOrderMgr : (I3525) Job Order File s3://proseo-data-inpuls/joborders/2026/6/9/7/f508e72c-9be2-432c-904e-f410dd62da7b.xml deleted from processing facility inpuls 2026-09-03T12:34:34.529Z INFO 1 --- [http-nio-8080-exec-18] d.d.p.ordermgr.rest.ProcessingOrderMgr : (I3554) Order with id 2,763,163 deleted 2026-09-03T12:34:34.530Z INFO 1 --- [http-nio-8080-exec-9] d.d.p.ordermgr.rest.ProcessingOrderMgr : (I3554) Order with id 2,758,748 deleted 2026-09-03T12:34:34.548Z TRACE 1 --- [http-nio-8080-exec-1] d.d.p.ordermgr.rest.ProcessingOrderMgr : >>> deleteJOF(2758983) 2026-09-03T12:34:34.569Z INFO 1 --- [http-nio-8080-exec-3] d.d.p.ordermgr.rest.ProcessingOrderMgr : (I3525) Job Order File s3://proseo-data-inpuls/joborders/2026/6/10/8/4f701838-6a79-42c2-966b-e4a657fd8b48.xml deleted from processing facility inpuls 2026-09-03T12:34:34.577Z INFO 1 --- [http-nio-8080-exec-3] d.d.p.ordermgr.rest.ProcessingOrderMgr : (I3554) Order with id 2,770,845 deleted 2026-09-03T12:34:34.603Z INFO 1 --- [http-nio-8080-exec-1] d.d.p.ordermgr.rest.ProcessingOrderMgr : (I3525) Job Order File s3://proseo-data-inpuls/joborders/2026/6/9/7/2a12d11d-cfe2-4c79-bdd0-4c74f24f575d.xml deleted from processing facility inpuls 2026-09-03T12:34:34.604Z INFO 1 --- [http-nio-8080-exec-1] d.d.p.ordermgr.rest.ProcessingOrderMgr : (I3554) Order with id 2,758,781 deleted

The GUI loops over the selected orders and call/start the deletion of each in the order manager.These actions run in parallel.
@tangobravo62 The transaction is handled as
@transactional(isolation = Isolation.REPEATABLE_READ)
Clould this be a reason therefore?
I'll try to reproduce the bug ...

[emelchinger]

Reproduced on local system with 150 orders, they were all deleted during bulk processing.
I try to reproduce on INPULS system.

[tangobravo62]

Further log analysis on INPULS system shows that the deletion requests to the Order Manager fail, but this is not logged in the Order Manager itself, but in the GUI:

(1) GUI prepares deletion request:

2026-09-03T12:46:03.426Z TRACE 1 --- [http-nio-8080-exec-8] d.dlr.proseo.ui.gui.GUIOrderController   : >>> setState(2659026, DELETE, model)
2026-09-03T12:46:03.426Z TRACE 1 --- [http-nio-8080-exec-8] d.d.proseo.ui.gui.service.OrderService   : URI http://proseo-order-mgr:8080/proseo/order-mgr/v0.1/orders/2659026
2026-09-03T12:46:03.426Z TRACE 1 --- [http-nio-8080-exec-8] d.d.proseo.ui.gui.service.OrderService   : Found authentication: de.dlr.proseo.ui.gui.GUIAuthenticationToken@23585c56
2026-09-03T12:46:03.426Z TRACE 1 --- [http-nio-8080-exec-8] d.d.proseo.ui.gui.service.OrderService   : ... with username proseo
2026-09-03T12:46:03.426Z TRACE 1 --- [http-nio-8080-exec-8] d.d.proseo.ui.gui.service.OrderService   : ... with password [protected]
2026-09-03T12:46:03.427Z TRACE 1 --- [http-nio-8080-exec-8] d.dlr.proseo.ui.gui.GUIOrderController   : DEREFFERED STRING: org.springframework.web.context.request.async.DeferredResult@4399cfa0

(2) Order Manager processes request, apparently correctly:

2026-09-03T12:46:03.559Z TRACE 1 --- [http-nio-8080-exec-11] d.d.p.ordermgr.rest.OrderControllerImpl  : >>> deleteOrderById(2659026)
2026-09-03T12:46:03.560Z TRACE 1 --- [http-nio-8080-exec-11] d.d.p.ordermgr.rest.ProcessingOrderMgr   : >>> deleteOrderById(2659026)
2026-09-03T12:46:03.561Z TRACE 1 --- [http-nio-8080-exec-11] d.d.p.ordermgr.rest.ProcessingOrderMgr   : >>> deleteOrder(UMAS_2026-05-27_rev08)
2026-09-03T12:46:03.562Z TRACE 1 --- [http-nio-8080-exec-11] d.d.p.ordermgr.rest.ProcessingOrderMgr   : >>> prepareOrderToDelete(UMAS_2026-05-27_rev08)
2026-09-03T12:46:03.566Z TRACE 1 --- [http-nio-8080-exec-11] d.d.p.ordermgr.rest.ProcessingOrderMgr   : >>> deleteJOF(2659674)
2026-09-03T12:46:03.566Z  INFO 1 --- [http-nio-8080-exec-11] d.d.p.ordermgr.rest.ProcessingOrderMgr   : (I3554) Order with id 2,659,026 deleted

(3) But GUI records not a 204 (NO CONTENT) status, as would be expected after a deletion, but a 304 (NOT MODIFIED) status, which signals a RuntimeException in the Order Manager:

2026-09-03T12:46:03.567Z TRACE 1 --- [reactor-http-epoll-7] d.d.proseo.ui.gui.service.OrderService   : response:304
2026-09-03T12:46:03.568Z TRACE 1 --- [reactor-http-epoll-7] d.dlr.proseo.ui.gui.GUIOrderController   : Now in Consumer::accept(<304 NOT_MODIFIED Not Modified,[Warning:"199 proseo-order-mgr org.hibernate.TransientPropertyValueException: Persistent instance of 'de.dlr.proseo.model.JobStep' references an unsaved transient instance of 'de.dlr.proseo.model.Job' (persist the transient instance before flushing) [de.dlr.proseo.model.JobStep.job -> de.dlr.proseo.model.Job]", X-Content-Type-Options:"nosniff", X-XSS-Protection:"0", X-Frame-Options:"DENY", Date:"Thu, 03 Sep 2026 12:46:03 GMT"]>)

Indeed there may be an issue with the transaction handling, but more likely due to the recorded error message is an issue we already observed a while ago, where processing orders could not be deleted due to spurious job steps existing in the database. At the time I prepared a sequence of database commands to delete those job steps (but I need to look them up).

The next check therefore should be whether there are more than one job steps per job in the affected orders in the database, but not visible in the GUI (which was the symptom the last time).

(Note that there are issues with error logging in both modules: The Order Manager should log the RuntimeException, since it catches it and converts it to a 304 HTTP status, and the GUI should not treat a 304 code as OK. And probably the Order Manager should have logged the error and returned a 500 INTERNAL SERVER ERROR status code in the first place.)

[tangobravo62]

Alas it is not the same issue as before. I checked the processing order cited above, and it indeed has 15 job steps for its single job, but that is OK, since the processing order requests 15 output product classes. So no database inconsistencies at this point at least.

Following is the output of the database investigation:

proseo=# select * from processing_order where id = 2659026;
   id    | version | actual_completion_time | estimated_completion_time |       eviction_time        |   execution_time    | has_failed_job_steps |      identifier       | input_file_name | input_sensing_start_time | input_sensing_stop_time | endpoint_password | endpoint_uri | endpoint_username | order_source | order_state | output_file_class | priority | processing_mode | product_retention_period | production_type | release_time | slice_duration | slice_overlap | slicing_type |     start_time      |     state_message     |      stop_time      | submission_time |                 uuid                 | mission_id | workflow_id | auto_close | auto_release | closing_time | input_data_timeout_period | on_input_data_timeout_fail | auto_generate_steps
---------+---------+------------------------+---------------------------+----------------------------+---------------------+----------------------+-----------------------+-----------------+--------------------------+-------------------------+-------------------+--------------+-------------------+--------------+-------------+-------------------+----------+-----------------+--------------------------+-----------------+--------------+----------------+---------------+--------------+---------------------+-----------------------+---------------------+-----------------+--------------------------------------+------------+-------------+------------+--------------+--------------+---------------------------+----------------------------+---------------------
 2659026 |       8 |                        |                           | 2026-06-11 19:08:23.268221 | 2026-05-27 11:00:00 | t                    | UMAS_2026-05-27_rev08 |                 |                          |                         |                   |              |                   | OTHER        | CLOSED      | NRTI              |       50 | NRTI            |                          | SYSTEMATIC      |              |                |             0 | NONE         | 2026-05-27 00:00:00 | production has failed | 2026-05-28 00:00:00 |                 | dc7971cf-b196-4e68-9485-e10dacb22efe |          1 |             | f          | f            |              |                           | t                          |
(1 row)

proseo=# select * from job where processing_order_id = 2659026;
   id    | version | has_failed_job_steps | job_state | priority |     start_time      |      stop_time      | orbit_id | processing_facility_id | processing_order_id
---------+---------+----------------------+-----------+----------+---------------------+---------------------+----------+------------------------+---------------------
 2659050 |       5 | t                    | CLOSED    |       50 | 2026-05-27 00:00:00 | 2026-05-28 00:00:00 |          |                    131 |             2659026
(1 row)

proseo=# select * from job_step where job_id = 2659050;
   id    | version | is_failed | job_order_filename | job_step_state | priority | processing_completion_time | processing_mode | processing_start_time | processing_std_err | processing_std_out | stderr_log_level | stdout_log_level | job_id
---------+---------+-----------+--------------------+----------------+----------+----------------------------+-----------------+-----------------------+--------------------+--------------------+------------------+------------------+---------
 2659674 |       5 | t         |                    | CLOSED         |       50 |                            | NRTI            |                       |                    |                    |                1 |                1 | 2659050
 2659713 |       5 | t         |                    | CLOSED         |       50 |                            | NRTI            |                       |                    |                    |                1 |                1 | 2659050
 2659710 |       5 | t         |                    | CLOSED         |       50 |                            | NRTI            |                       |                    |                    |                1 |                1 | 2659050
 2659707 |       5 | t         |                    | CLOSED         |       50 |                            | NRTI            |                       |                    |                    |                1 |                1 | 2659050
 2659689 |       5 | t         |                    | CLOSED         |       50 |                            | NRTI            |                       |                    |                    |                1 |                1 | 2659050
 2659671 |       5 | t         |                    | CLOSED         |       50 |                            | NRTI            |                       |                    |                    |                1 |                1 | 2659050
 2659695 |       5 | t         |                    | CLOSED         |       50 |                            | NRTI            |                       |                    |                    |                1 |                1 | 2659050
 2659686 |       5 | t         |                    | CLOSED         |       50 |                            | NRTI            |                       |                    |                    |                1 |                1 | 2659050
 2659677 |       5 | t         |                    | CLOSED         |       50 |                            | NRTI            |                       |                    |                    |                1 |                1 | 2659050
 2659668 |       5 | t         |                    | CLOSED         |       50 |                            | NRTI            |                       |                    |                    |                1 |                1 | 2659050
 2659683 |       5 | t         |                    | CLOSED         |       50 |                            | NRTI            |                       |                    |                    |                1 |                1 | 2659050
 2659692 |       5 | t         |                    | CLOSED         |       50 |                            | NRTI            |                       |                    |                    |                1 |                1 | 2659050
 2659701 |       5 | t         |                    | CLOSED         |       50 |                            | NRTI            |                       |                    |                    |                1 |                1 | 2659050
 2659665 |       5 | t         |                    | CLOSED         |       50 |                            | NRTI            |                       |                    |                    |                1 |                1 | 2659050
 2659704 |       5 | t         |                    | CLOSED         |       50 |                            | NRTI            |                       |                    |                    |                1 |                1 | 2659050
(15 rows)

The CLI says:

prosEO (S5P)> order show -v UMAS_2026-05-27_rev08
---
id: 2659026
version: 8
missionCode: "S5P"
identifier: "UMAS_2026-05-27_rev08"
uuid: "dc7971cf-b196-4e68-9485-e10dacb22efe"
priority: 50
orderState: "CLOSED"
[...]
requestedProductClasses:
- "L3__AI____"
- "L3__ALH___"
- "L3__AOD___"
- "L3__ASSA__"
- "L3__CA_CRB"
- "L3__CF_CRB"
- "L3__CF____"
- "L3__CH_CRB"
- "L3__COT___"
- "L3__CTH___"
- "L3__HCHO__"
- "L3__O3____"
- "L3__SO2LH_"
- "L3__SO2___"
- "L3__UVI___"
[...]
jobStepStates:
- "CLOSED"
[...]