cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

MarsSu
by New Contributor II
  • 3155 Views
  • 5 replies
  • 1 kudos

Resolved! Databricks job about spark structured streaming zero downtime deployment in terraform.

I would like to ask how to implement zero downtime deployment of spark structured streaming in databricks job compute with terraform. Because we will upgrade spark application code version. But currently we found every deployment will cancel original...

  • 3155 Views
  • 5 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

@Mars Su​ :Yes, you can implement zero downtime deployment of Spark Structured Streaming in Databricks job compute using Terraform. One way to achieve this is by using Databricks' "job clusters" feature, which allows you to create a cluster specifica...

  • 1 kudos
4 More Replies
pvm26042000
by New Contributor III
  • 424 Views
  • 1 replies
  • 2 kudos

I am using Spark SQL to import their data into a machine learning pipeline. Once data is imported I want performs machine learning tasks using Spark...

I am using Spark SQL to import their data into a machine learning pipeline. Once data is imported I want performs machine learning tasks using Spark ML. So I should use what compute tools is best suited for this use case? Please help me!!! Thank y...

  • 424 Views
  • 1 replies
  • 2 kudos
Latest Reply
Debayan
Esteemed Contributor III
  • 2 kudos

Hi, please refer https://docs.databricks.com/machine-learning/index.html, please let us know if this helps.

  • 2 kudos
Jfoxyyc
by Valued Contributor
  • 774 Views
  • 2 replies
  • 4 kudos

Databricks Terraform - how to manage databricks entirely through Terraform?

I'm stuck at a point where I can't automatically set up everything about a databricks environment due to the fact that service principals can't be made an admin at the account level (accounts.azuredatabricks.net, similar for aws). Going into a bare t...

  • 774 Views
  • 2 replies
  • 4 kudos
Latest Reply
daniel_sahal
Esteemed Contributor
  • 4 kudos

Unfortunately there are still some limitations with doing IaC on Databricks with Terraform (ex. another one is that you can't setup KeyVault as a secret store with Service Principal).I think that instead of doing stuff manually, you can authenticate ...

  • 4 kudos
1 More Replies
User16826994223
by Honored Contributor III
  • 251 Views
  • 0 replies
  • 0 kudos

Data scientist Job Profile will be relevant in the future? By seeing current features in Databricks like AUTO ML, I am assuming that should the Data ...

Data scientist Job Profile will be relevant in the future?By seeing current features in Databricks like AUTO ML, I am assuming that should the Data scientist job will be mostly automated and sooner the data scientist in the company will start decl...

  • 251 Views
  • 0 replies
  • 0 kudos
Labels