- 2513 Views
- 1 replies
- 2 kudos
How to choose the right node type for my Databricks workload?
Hey all, I’m setting up a Databricks workspace to run data pipelines + ML training. I’m unsure which node types to pick (driver vs worker, instance size, memory vs CPU optimized) for different workloads. For example small ETL jobs vs large batch proc...
- 2513 Views
- 1 replies
- 2 kudos
- 2 kudos
Hi @Ednexllc ,It really depends on many factor like workload type, size of your data, number of tables etc. You can check some recommendations given by Databricks here:Compute configuration recommendations | Databricks on AWSAnd also here - you shoul...
- 2 kudos
- 2193 Views
- 1 replies
- 0 kudos
Terraform error deploying Databricks Asset Bundle
Hi all ,I am deploying a very simple DAB from an Azure DevOps Pipeline with an self hosted agent, there is no error messages but in the Databricks workspace there is nothing deployed and the files of the bundle are uploaded in the workspace. When I p...
- 2193 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @armandovl ,Try pointing your Databricks bundle to a system-installed Terraform binary. In your bundle config file, configure it to use the system’s Terraform instead of the bundled one.Also, ensure the Terraform binary is available in the agent’s...
- 0 kudos
- 3888 Views
- 1 replies
- 0 kudos
problem with workspace after metastore deleted
I am completely new to Databricks AWS and start working on it a week ago. Pls excuse me if I ask or did something silly.I created a workspace and a single node cluster for testing. A metastore was created from Databricks quickstart and it was automa...
- 3888 Views
- 1 replies
- 0 kudos
- 0 kudos
I restarted the compute node and this problem went away.ErrorClass=METASTORE_DOES_NOT_EXIST] Metastore 'b11fb1a0-a462-4dfb-b91b-e0795fde10b0' does not exist.New question: I am testing Databricks with non-AWS S3 object storage. I can access the non-A...
- 0 kudos