cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
Sulfikkar
by • Contributor
  • 6964 Views
  • 8 replies
  • 4 kudos

Cluster Scoped init script through pulumi

I am trying to run a cluster-scoped init script through Pulumi. I have referred to this documentation https://learn.microsoft.com/en-us/azure/databricks/clusters/configure#spark-configuration However, looks like the documentation is not very clear.I ...

Data Engineering
Reply
  • 6964 Views
  • 8 replies
  • 4 kudos
View Replies
Latest Reply
Vivian_Wilfred
Databricks Employee
  • 4 kudos

Hi @Sulfikkar Basheer Shylaja​ , Why don't you store the init-script on DBFS and just pass the dbfs:/ path of the init script in Pulumi? You could just run this code on a notebook-%python dbutils.fs.put("/databricks/init-scripts/set-private-pip-repos...

  • 4 kudos
7 More Replies
Jonas89
by • New Contributor
  • 7854 Views
  • 2 replies
  • 0 kudos

Databricks Devops Release Pipeline Abort

We've built a release pipeline to our Databricks Workspaces, using the VNET Template. It's working end-to-end but intermittent aborts are occurring when workspace is recreated.For example, 4th of April (Monday) We recreated the workspaces and no abor...

1_PipelineError 2_PipelineError 3_PipelineError 4_PipelineError
Data Engineering
Reply
  • 7854 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Jonas Oliveira de Souza​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that be...

  • 0 kudos
1 More Replies
Chris_Shehu
by • Valued Contributor III
  • 3812 Views
  • 1 replies
  • 5 kudos

Resolved! Getting errors while following Microsoft Databricks Best-Practices for DevOps Integration

I'm currently trying to follow the Software engineering best practices for notebooks - Azure Databricks guide but I keep running into the following during step 4.5: Run the test============================= test session starts =======================...

image.png image image image
Data Engineering
Reply
  • 3812 Views
  • 1 replies
  • 5 kudos
View Replies
Latest Reply
Chris_Shehu
Valued Contributor III
  • 5 kudos

Closing the loop on this in case anyone gets stuck in the same situation. You can see in the images that the transforms_test.py shows a different icon then the testdata.csv. This is because it was saved as a juypter notebook not a .py file. When the ...

  • 5 kudos
MadelynM
by • Databricks Employee
  • 1009 Views
  • 0 replies
  • 1 kudos

vimeo.com

Repos let you use Git functionality such as cloning a remote repo, managing branches, pushing and pulling changes and visually comparing differences upon commit. Here's a quick video (3:56) on setting up a repo for Databricks on AWS. Pre-reqs: Git in...

Data Engineering
Reply
  • 1009 Views
  • 0 replies
  • 1 kudos
SEOCO
by • New Contributor II
  • 2772 Views
  • 3 replies
  • 3 kudos

Passing parameters from DevOps Pipeline/release to DataBricks Notebook

Hi,This is all a bit new to me.Does anybody have any idea how to pass a parameter to the Databricks notebook.I have a DevOps pipeline/release that moves my databricks notebooks towards QA and Production environment. The only problem I am facing is th...

Data Engineering
Reply
  • 2772 Views
  • 3 replies
  • 3 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 3 kudos

@Mario Walle​ - If @Hubert Dudek​'s answer solved the issue, would you be happy to mark his answer as best so that it will be more visible to other members?

  • 3 kudos
2 More Replies
User16137833804
by • Databricks Employee
  • 1415 Views
  • 1 replies
  • 1 kudos

Once I set up the Git Server Proxy, what would be the best way to set alerts in case the Cluster Proxy goes down?

Data Engineering
Reply
  • 1415 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
sajith_appukutt
Honored Contributor II
  • 1 kudos

You could have the single node cluster where proxy is installed monitored by one of the tools like cloudwatch, azure monitor, datadog etc and have it configured to send alerts on node failure

  • 1 kudos
Labels