Skip to content

updates compute clusters used #298

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
tomjemmett wants to merge 1 commit into from
Open

updates compute clusters used #298

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 5 additions & 4 deletions databricks_workflows/nhp_data-ecds.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,22 +48,23 @@ resources:
job_clusters:
- job_cluster_key: nhp_data
new_cluster:
spark_version: 17.3.x-scala2.13
spark_version: 18.x-scala2.13
spark_conf:
spark.databricks.delta.properties.defaults.autoOptimize.optimizeWrite: "true"
spark.databricks.delta.properties.defaults.autoOptimize.autoCompact: "true"
spark.databricks.sql.initial.catalog.namespace: udal_lake_mart
azure_attributes:
availability: ON_DEMAND_AZURE
spot_bid_max_price: 100
node_type_id: Standard_D4ds_v5
node_type_id: Standard_D8ds_v5
driver_node_type_id: Standard_D8ds_v5
custom_tags:
Application: UDAL
Workload_Type: Job
Workload_Size: Small
Workload_Size: Medium
spark_env_vars:
PYSPARK_PYTHON: /databricks/python3/bin/python3
policy_id: 000816B2986C0B29
policy_id: 0001CA9D7F08DD46
data_security_mode: USER_ISOLATION
runtime_engine: STANDARD
kind: CLASSIC_PREVIEW
Expand Down
21 changes: 11 additions & 10 deletions databricks_workflows/nhp_data-extract_nhp_for_containers.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -178,29 +178,30 @@ resources:
job_clusters:
- job_cluster_key: nhp_data
new_cluster:
spark_version: 17.3.x-scala2.13
spark_version: 18.x-scala2.13
spark_conf:
spark.databricks.sql.initial.catalog.namespace: udal_lake_mart
spark.databricks.delta.properties.defaults.autoOptimize.optimizeWrite: "true"
spark.databricks.delta.properties.defaults.autoOptimize.autoCompact: "true"
spark.databricks.sql.initial.catalog.namespace: udal_lake_mart
spark.master: local[*, 4]
spark.databricks.cluster.profile: singleNode
azure_attributes:
availability: ON_DEMAND_AZURE
spot_bid_max_price: 100
node_type_id: Standard_D4ds_v5
node_type_id: Standard_E8ads_v7
custom_tags:
Application: UDAL
Workload_Type: Job
Workload_Size: Small
Workload_Size: Medium
ResourceClass: SingleNode
spark_env_vars:
PYSPARK_PYTHON: /databricks/python3/bin/python3
policy_id: 000816B2986C0B29
data_security_mode: USER_ISOLATION
policy_id: 001D6F598E9C0B76
data_security_mode: SINGLE_USER
runtime_engine: STANDARD
kind: CLASSIC_PREVIEW
is_single_node: false
autoscale:
min_workers: 1
max_workers: 4
is_single_node: true
num_workers: 0
tags:
group: nhp_data_model
env: ${bundle.target}
Expand Down
9 changes: 5 additions & 4 deletions databricks_workflows/nhp_data-hes-aae.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,22 +53,23 @@ resources:
job_clusters:
- job_cluster_key: nhp_data
new_cluster:
spark_version: 17.3.x-scala2.13
spark_version: 18.x-scala2.13
spark_conf:
spark.databricks.delta.properties.defaults.autoOptimize.optimizeWrite: "true"
spark.databricks.delta.properties.defaults.autoOptimize.autoCompact: "true"
spark.databricks.sql.initial.catalog.namespace: udal_lake_mart
azure_attributes:
availability: ON_DEMAND_AZURE
spot_bid_max_price: 100
node_type_id: Standard_D4ds_v5
node_type_id: Standard_D8ds_v5
driver_node_type_id: Standard_D8ds_v5
custom_tags:
Application: UDAL
Workload_Type: Job
Workload_Size: Small
Workload_Size: Medium
spark_env_vars:
PYSPARK_PYTHON: /databricks/python3/bin/python3
policy_id: 000816B2986C0B29
policy_id: 0001CA9D7F08DD46
data_security_mode: USER_ISOLATION
runtime_engine: STANDARD
kind: CLASSIC_PREVIEW
Expand Down
9 changes: 5 additions & 4 deletions databricks_workflows/nhp_data-hes-apc.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,22 +53,23 @@ resources:
job_clusters:
- job_cluster_key: nhp_data
new_cluster:
spark_version: 17.3.x-scala2.13
spark_version: 18.x-scala2.13
spark_conf:
spark.databricks.delta.properties.defaults.autoOptimize.optimizeWrite: "true"
spark.databricks.delta.properties.defaults.autoOptimize.autoCompact: "true"
spark.databricks.sql.initial.catalog.namespace: udal_lake_mart
azure_attributes:
availability: ON_DEMAND_AZURE
spot_bid_max_price: 100
node_type_id: Standard_D4ds_v5
node_type_id: Standard_D8ds_v5
driver_node_type_id: Standard_D8ds_v5
custom_tags:
Application: UDAL
Workload_Type: Job
Workload_Size: Small
Workload_Size: Medium
spark_env_vars:
PYSPARK_PYTHON: /databricks/python3/bin/python3
policy_id: 000816B2986C0B29
policy_id: 0001CA9D7F08DD46
data_security_mode: USER_ISOLATION
runtime_engine: STANDARD
kind: CLASSIC_PREVIEW
Expand Down
9 changes: 5 additions & 4 deletions databricks_workflows/nhp_data-hes-opa.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,22 +43,23 @@ resources:
job_clusters:
- job_cluster_key: nhp_data
new_cluster:
spark_version: 17.3.x-scala2.13
spark_version: 18.x-scala2.13
spark_conf:
spark.databricks.delta.properties.defaults.autoOptimize.optimizeWrite: "true"
spark.databricks.delta.properties.defaults.autoOptimize.autoCompact: "true"
spark.databricks.sql.initial.catalog.namespace: udal_lake_mart
azure_attributes:
availability: ON_DEMAND_AZURE
spot_bid_max_price: 100
node_type_id: Standard_D4ds_v5
node_type_id: Standard_D8ds_v5
driver_node_type_id: Standard_D8ds_v5
custom_tags:
Application: UDAL
Workload_Type: Job
Workload_Size: Small
Workload_Size: Medium
spark_env_vars:
PYSPARK_PYTHON: /databricks/python3/bin/python3
policy_id: 000816B2986C0B29
policy_id: 0001CA9D7F08DD46
data_security_mode: USER_ISOLATION
runtime_engine: STANDARD
kind: CLASSIC_PREVIEW
Expand Down
9 changes: 5 additions & 4 deletions databricks_workflows/nhp_data-inpatients.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,22 +39,23 @@ resources:
job_clusters:
- job_cluster_key: nhp_data
new_cluster:
spark_version: 17.3.x-scala2.13
spark_version: 18.x-scala2.13
spark_conf:
spark.databricks.delta.properties.defaults.autoOptimize.optimizeWrite: "true"
spark.databricks.delta.properties.defaults.autoOptimize.autoCompact: "true"
spark.databricks.sql.initial.catalog.namespace: udal_lake_mart
azure_attributes:
availability: ON_DEMAND_AZURE
spot_bid_max_price: 100
node_type_id: Standard_D4ds_v5
node_type_id: Standard_D8ds_v5
driver_node_type_id: Standard_D8ds_v5
custom_tags:
Application: UDAL
Workload_Type: Job
Workload_Size: Small
Workload_Size: Medium
spark_env_vars:
PYSPARK_PYTHON: /databricks/python3/bin/python3
policy_id: 000816B2986C0B29
policy_id: 0001CA9D7F08DD46
data_security_mode: USER_ISOLATION
runtime_engine: STANDARD
kind: CLASSIC_PREVIEW
Expand Down
9 changes: 5 additions & 4 deletions databricks_workflows/nhp_data-inputs-lad23cd.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,22 +67,23 @@ resources:
job_clusters:
- job_cluster_key: nhp_data
new_cluster:
spark_version: 17.3.x-scala2.13
spark_version: 18.x-scala2.13
spark_conf:
spark.databricks.delta.properties.defaults.autoOptimize.optimizeWrite: "true"
spark.databricks.delta.properties.defaults.autoOptimize.autoCompact: "true"
spark.databricks.sql.initial.catalog.namespace: udal_lake_mart
azure_attributes:
availability: ON_DEMAND_AZURE
spot_bid_max_price: 100
node_type_id: Standard_D4ds_v5
node_type_id: Standard_D8ds_v5
driver_node_type_id: Standard_D8ds_v5
custom_tags:
Application: UDAL
Workload_Type: Job
Workload_Size: Small
Workload_Size: Medium
spark_env_vars:
PYSPARK_PYTHON: /databricks/python3/bin/python3
policy_id: 000816B2986C0B29
policy_id: 0001CA9D7F08DD46
data_security_mode: USER_ISOLATION
runtime_engine: STANDARD
kind: CLASSIC_PREVIEW
Expand Down
9 changes: 5 additions & 4 deletions databricks_workflows/nhp_data-inputs-provider.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -100,22 +100,23 @@ resources:
job_clusters:
- job_cluster_key: nhp_data
new_cluster:
spark_version: 17.3.x-scala2.13
spark_version: 18.x-scala2.13
spark_conf:
spark.databricks.delta.properties.defaults.autoOptimize.optimizeWrite: "true"
spark.databricks.delta.properties.defaults.autoOptimize.autoCompact: "true"
spark.databricks.sql.initial.catalog.namespace: udal_lake_mart
azure_attributes:
availability: ON_DEMAND_AZURE
spot_bid_max_price: 100
node_type_id: Standard_D4ds_v5
node_type_id: Standard_D8ds_v5
driver_node_type_id: Standard_D8ds_v5
custom_tags:
Application: UDAL
Workload_Type: Job
Workload_Size: Small
Workload_Size: Medium
spark_env_vars:
PYSPARK_PYTHON: /databricks/python3/bin/python3
policy_id: 000816B2986C0B29
policy_id: 0001CA9D7F08DD46
data_security_mode: USER_ISOLATION
runtime_engine: STANDARD
kind: CLASSIC_PREVIEW
Expand Down
9 changes: 5 additions & 4 deletions databricks_workflows/nhp_data-outpatients.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,22 +48,23 @@ resources:
job_clusters:
- job_cluster_key: nhp_data
new_cluster:
spark_version: 17.3.x-scala2.13
spark_version: 18.x-scala2.13
spark_conf:
spark.databricks.delta.properties.defaults.autoOptimize.optimizeWrite: "true"
spark.databricks.delta.properties.defaults.autoOptimize.autoCompact: "true"
spark.databricks.sql.initial.catalog.namespace: udal_lake_mart
azure_attributes:
availability: ON_DEMAND_AZURE
spot_bid_max_price: 100
node_type_id: Standard_D4ds_v5
node_type_id: Standard_D8ds_v5
driver_node_type_id: Standard_D8ds_v5
custom_tags:
Application: UDAL
Workload_Type: Job
Workload_Size: Small
Workload_Size: Medium
spark_env_vars:
PYSPARK_PYTHON: /databricks/python3/bin/python3
policy_id: 000816B2986C0B29
policy_id: 0001CA9D7F08DD46
data_security_mode: USER_ISOLATION
runtime_engine: STANDARD
kind: CLASSIC_PREVIEW
Expand Down
9 changes: 5 additions & 4 deletions databricks_workflows/nhp_data-population_projections.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -64,22 +64,23 @@ resources:
job_clusters:
- job_cluster_key: nhp_data
new_cluster:
spark_version: 17.3.x-scala2.13
spark_version: 18.x-scala2.13
spark_conf:
spark.databricks.delta.properties.defaults.autoOptimize.optimizeWrite: "true"
spark.databricks.delta.properties.defaults.autoOptimize.autoCompact: "true"
spark.databricks.sql.initial.catalog.namespace: udal_lake_mart
azure_attributes:
availability: ON_DEMAND_AZURE
spot_bid_max_price: 100
node_type_id: Standard_D4ds_v5
node_type_id: Standard_D8ds_v5
driver_node_type_id: Standard_D8ds_v5
custom_tags:
Application: UDAL
Workload_Type: Job
Workload_Size: Small
Workload_Size: Medium
spark_env_vars:
PYSPARK_PYTHON: /databricks/python3/bin/python3
policy_id: 000816B2986C0B29
policy_id: 0001CA9D7F08DD46
data_security_mode: USER_ISOLATION
runtime_engine: STANDARD
kind: CLASSIC_PREVIEW
Expand Down
21 changes: 11 additions & 10 deletions databricks_workflows/nhp_data-reference_data.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -97,29 +97,30 @@ resources:
job_clusters:
- job_cluster_key: nhp_data
new_cluster:
spark_version: 17.3.x-scala2.13
spark_version: 18.x-scala2.13
spark_conf:
spark.databricks.sql.initial.catalog.namespace: udal_lake_mart
spark.databricks.delta.properties.defaults.autoOptimize.optimizeWrite: "true"
spark.databricks.delta.properties.defaults.autoOptimize.autoCompact: "true"
spark.databricks.sql.initial.catalog.namespace: udal_lake_mart
spark.master: local[*, 4]
spark.databricks.cluster.profile: singleNode
azure_attributes:
availability: ON_DEMAND_AZURE
spot_bid_max_price: 100
node_type_id: Standard_D4ds_v5
node_type_id: Standard_E8ads_v7
custom_tags:
Application: UDAL
Workload_Type: Job
Workload_Size: Small
Workload_Size: Medium
ResourceClass: SingleNode
spark_env_vars:
PYSPARK_PYTHON: /databricks/python3/bin/python3
policy_id: 000816B2986C0B29
data_security_mode: USER_ISOLATION
policy_id: 001D6F598E9C0B76
data_security_mode: SINGLE_USER
runtime_engine: STANDARD
kind: CLASSIC_PREVIEW
is_single_node: false
autoscale:
min_workers: 1
max_workers: 4
is_single_node: true
num_workers: 0
tags:
group: nhp_data
env: ${bundle.target}
Expand Down
Loading