cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
MarsSu
by • New Contributor II
  • 8437 Views
  • 5 replies
  • 1 kudos

Resolved! Databricks job about spark structured streaming zero downtime deployment in terraform.

I would like to ask how to implement zero downtime deployment of spark structured streaming in databricks job compute with terraform. Because we will upgrade spark application code version. But currently we found every deployment will cancel original...

Data Engineering
Reply
  • 8437 Views
  • 5 replies
  • 1 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 1 kudos

@Mars Su​ :Yes, you can implement zero downtime deployment of Spark Structured Streaming in Databricks job compute using Terraform. One way to achieve this is by using Databricks' "job clusters" feature, which allows you to create a cluster specifica...

  • 1 kudos
4 More Replies
pvm26042000
by • New Contributor III
  • 752 Views
  • 1 replies
  • 2 kudos

I am using Spark SQL to import their data into a machine learning pipeline. Once data is imported I want performs machine learning tasks using Spark...

I am using Spark SQL to import their data into a machine learning pipeline. Once data is imported I want performs machine learning tasks using Spark ML. So I should use what compute tools is best suited for this use case? Please help me!!! Thank y...

Data Engineering
Reply
  • 752 Views
  • 1 replies
  • 2 kudos
View Replies
Latest Reply
Debayan
Esteemed Contributor III
  • 2 kudos

Hi, please refer https://docs.databricks.com/machine-learning/index.html, please let us know if this helps.

  • 2 kudos
Jfoxyyc
by • Valued Contributor
  • 1535 Views
  • 2 replies
  • 4 kudos

Databricks Terraform - how to manage databricks entirely through Terraform?

I'm stuck at a point where I can't automatically set up everything about a databricks environment due to the fact that service principals can't be made an admin at the account level (accounts.azuredatabricks.net, similar for aws). Going into a bare t...

Data Engineering
Reply
  • 1535 Views
  • 2 replies
  • 4 kudos
View Replies
Latest Reply
daniel_sahal
Esteemed Contributor
  • 4 kudos

Unfortunately there are still some limitations with doing IaC on Databricks with Terraform (ex. another one is that you can't setup KeyVault as a secret store with Service Principal).I think that instead of doing stuff manually, you can authenticate ...

  • 4 kudos
1 More Replies
User16826994223
by • Honored Contributor III
  • 403 Views
  • 0 replies
  • 0 kudos

Data scientist Job Profile will be relevant in the future? By seeing current features in Databricks like AUTO ML, I am assuming that should the Data ...

Data scientist Job Profile will be relevant in the future?By seeing current features in Databricks like AUTO ML, I am assuming that should the Data scientist job will be mostly automated and sooner the data scientist in the company will start decl...

Data Engineering
Reply
  • 403 Views
  • 0 replies
  • 0 kudos
Labels