Repository navigation
updates compute clusters used - #298
tomjemmett wants to merge 1 commit into
Conversation
- switches to spark 18 (current lts) - switches some workflows to use the medium policy - switches other policies to use single node policy
There was a problem hiding this comment.
🔵 Needs a closer look
Correctness depends on external Databricks workspace state (opaque policy IDs, DBR 18 runtime availability, and the SINGLE_USER/USER_ISOLATION access-mode changes) that cannot be verified here and warrants human confirmation.
No open issues 🎉
What changed in this PR
This PR updates the Databricks compute cluster definitions used by the nhp_data asset-bundle workflows. It bumps the Databricks Runtime to 18.x-scala2.13 (described as the current LTS) and reorganises how each job's cluster is sized and governed, splitting the workflows into two tiers: lightweight orchestration/extract jobs move to a single-node cluster, while the data-processing jobs move to a larger "medium" multi-node cluster.
Changes:
- Upgrade
spark_versionfrom17.3.x-scala2.13to18.x-scala2.13across all 11 workflow cluster definitions. - Convert
reference_dataandextract_nhp_for_containersto single-node clusters (Standard_E8ads_v7,singleNodeprofile,num_workers: 0,SINGLE_USER, new policy001D6F598E9C0B76). - Move the remaining data jobs to medium multi-node clusters (
Standard_D8ds_v5worker + driver,Workload_Size: Medium, new policy0001CA9D7F08DD46), retainingUSER_ISOLATIONand 1–4 worker autoscaling.
| File | Description |
|---|---|
| databricks_workflows/nhp_data-reference_data.yaml | Converts cluster to single-node (E8ads_v7, SINGLE_USER, single-node policy) on DBR 18 |
| databricks_workflows/nhp_data-extract_nhp_for_containers.yaml | Converts cluster to single-node (E8ads_v7, SINGLE_USER, single-node policy) on DBR 18 |
| databricks_workflows/nhp_data-population_projections.yaml | Medium multi-node cluster (D8ds_v5 + driver, medium policy) on DBR 18 |
| databricks_workflows/nhp_data-outpatients.yaml | Medium multi-node cluster (D8ds_v5 + driver, medium policy) on DBR 18 |
| databricks_workflows/nhp_data-inputs-provider.yaml | Medium multi-node cluster (D8ds_v5 + driver, medium policy) on DBR 18 |
| databricks_workflows/nhp_data-inputs-lad23cd.yaml | Medium multi-node cluster (D8ds_v5 + driver, medium policy) on DBR 18 |
| databricks_workflows/nhp_data-inpatients.yaml | Medium multi-node cluster (D8ds_v5 + driver, medium policy) on DBR 18 |
| databricks_workflows/nhp_data-hes-opa.yaml | Medium multi-node cluster (D8ds_v5 + driver, medium policy) on DBR 18 |
| databricks_workflows/nhp_data-hes-apc.yaml | Medium multi-node cluster (D8ds_v5 + driver, medium policy) on DBR 18 |
| databricks_workflows/nhp_data-hes-aae.yaml | Medium multi-node cluster (D8ds_v5 + driver, medium policy) on DBR 18 |
| databricks_workflows/nhp_data-ecds.yaml | Medium multi-node cluster (D8ds_v5 + driver, medium policy) on DBR 18 |
I reviewed all 11 changed files. The two single-node definitions are internally consistent with each other (correct canonical single-node setup: spark.master: local[*, 4], singleNode profile, is_single_node: true, num_workers: 0, no leftover autoscale, and SINGLE_USER access mode which is required for single-node Unity Catalog clusters), and the nine medium definitions are consistent with each other (matching node type, driver node type, policy ID, retained USER_ISOLATION and autoscaling). I found no concrete code-level defects to comment on.
Review effort: Balanced
Stack created with GitHub Stacks CLI • Give Feedback 💬