- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-11-2022 10:09 PM
I am not v clear on the question but when we provision DLT workflow you can tweak what type of cluster and the servers etc EG
{
"clusters": [
{
"label": "default",
"node_type_id": "c5.4xlarge",
"driver_node_type_id": "c5.4xlarge",
"num_workers": 20,
"spark_conf": {
"spark.databricks.io.parquet.nativeReader.enabled": "false"
},
"aws_attributes": {
"instance_profile_arn": "arn:aws:..."
}
},
{
"label": "maintenance",
"aws_attributes": {
"instance_profile_arn": "arn:aws:..."
}
}
]
}
it will then provision those servers in your cloud provider
So the total cost will be the DBU cost and the cost of your EC2 instances