[tomjemmett]

limitations in udal prevent the current compute clusters from having more than two workflows running at once.

run the op after the a&e data, as these together take roughly same time
as ip data.

also run the inputs/tpma explorer for provider/local authorities in parallel

---

_{Stack created with GitHub Stacks CLI • Give Feedback 💬}

[Copilot]

🔵 Needs a closer look

The resulting DAG serialises IP after A&E, which appears to diverge from the description's stated parallelisation goal, and confirming the intended production scheduling requires domain knowledge of job runtimes.

1 open issue

• IP now waits on A&E, reducing pipeline parallelism · New

What changed in this PR

This PR reorganises the Generate NHP Data Databricks workflow DAG to work around a UDAL constraint that allows at most two child workflows to run concurrently on the current compute cluster. It reshapes the task dependencies so A&E (ECDS) runs before IP/OP, runs the provider and local-authority inputs jobs in parallel, and chains the container extract job onto the end of the pipeline.

Changes:

• generate_ip and generate_op now depend on generate_ecds instead of generate_nhp_reference_data.
• generate_inputs_data_local_authorities now depends on generate_ip/generate_op (instead of generate_inputs_data_provider), so provider and LA inputs run in parallel; the redundant generate_ecds dependency was dropped from the provider inputs task.
• Added a new extract_data task (Extract_NHP_for_containers) that runs after both inputs jobs complete.

File	Description
databricks_workflows/​nhp_data.yaml	Re-wires the task dependency graph to cap concurrency at two workflows, parallelises the inputs jobs, and appends the container extract job.

_{Review effort: Balanced}