cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

timothy_uk
by New Contributor III
  • 1130 Views
  • 1 replies
  • 1 kudos

Mysterious simultaneous long-running Databricks Workflows

Hi,This happened across 4x seemingly unrelated workflows at the same time of the day - all 4x workflows eventually completed successfully. It appeared that all workflows sat idling despite triggering via the Jobs API. The two symptoms I have observed...

  • 1130 Views
  • 1 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @Timothy Lin​ Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question. Thanks.

  • 1 kudos
Oliver_Angelil
by Valued Contributor II
  • 1868 Views
  • 2 replies
  • 2 kudos

Automated CI code checks using workflows when PR is raised

I'm familiar with Github Actions workflows to automate code checks whenever a PR is raised to a specified branch. For example for Python code, very useful is if unit tests (e.g. pytest), syntax (flake8), and code formatting (black formatter), type h...

  • 1868 Views
  • 2 replies
  • 2 kudos
Latest Reply
Priyag1
Honored Contributor II
  • 2 kudos

In a typical software development workflow (e.g. Github flow), a feature branch is created based on the master branch for feature development. A notebook can be synced to the feature branch via Github integration. Or a notebook can be exported from D...

  • 2 kudos
1 More Replies
youssefmrini
by Databricks Employee
  • 2002 Views
  • 1 replies
  • 0 kudos
  • 2002 Views
  • 1 replies
  • 0 kudos
Latest Reply
youssefmrini
Databricks Employee
  • 0 kudos

Feel free to read my medium blog where I summarized all the featureshttps://medium.com/@youssefmrini/databricks-workflows-features-oriented-3f9ec025301a

  • 0 kudos
Hubert-Dudek
by Esteemed Contributor III
  • 1366 Views
  • 1 replies
  • 21 kudos

Databricks Workflows is introducing a new feature called "Task Values", a simple API for setting and retrieving small values from tasks. Tas...

Databricks Workflows is introducing a new feature called "Task Values", a simple API for setting and retrieving small values from tasks. Tasks can now output values that can be referenced in subsequent tasks, making it easier to create more expressiv...

db-243-blog-img-1
  • 1366 Views
  • 1 replies
  • 21 kudos
Latest Reply
newforesee
New Contributor II
  • 21 kudos

We urgently hope for this feature, but to date, we have found that it is only available in Python. Do you have any plans to support Scala?Please provide more support for Scala.

  • 21 kudos
pvignesh92
by Honored Contributor
  • 1173 Views
  • 1 replies
  • 0 kudos

Resolved! How to use caller's rights in Databricks Workflows?

I have created a job in Databricks Workflow that runs a notebook and have given manage run permissions to users. If any user trigger the job, I expect the job to show their username in Run As Option as it was triggered by them and they have Manage Ru...

  • 1173 Views
  • 1 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Vigneshraja Palaniraj​ Rus as is a private preview feature. Please complete this sign up form which will be reviewed by PM and you'll receive the confirmation once it's enabled.

  • 0 kudos
youssefmrini
by Databricks Employee
  • 1281 Views
  • 1 replies
  • 2 kudos
  • 1281 Views
  • 1 replies
  • 2 kudos
Latest Reply
youssefmrini
Databricks Employee
  • 2 kudos

Continuous Jobs trigger mode has been introduced recently. To prevent unnecessary resource usage and reduce cost, Databricks automatically pauses a continuous job if there are more than five consecutive failures within a 24 hour period.

  • 2 kudos
Dataengineer_mm
by New Contributor
  • 1864 Views
  • 2 replies
  • 0 kudos

Databricks workflow migration to higher environments

How do we migrate the databricks workflows to higher environment ? I do see an option for calling the tasks (notebooks,python) from the github repositories. But as such how do we migrate the entire workflow jobs to other environment ?

  • 1864 Views
  • 2 replies
  • 0 kudos
Latest Reply
jose_gonzalez
Databricks Employee
  • 0 kudos

Hi @Menaka Murugesan​,Just a friendly follow-up. Did any of the responses help you to resolve your question? if it did, please mark it as best. Otherwise, please let us know if you still need help.

  • 0 kudos
1 More Replies
youssefmrini
by Databricks Employee
  • 1208 Views
  • 1 replies
  • 1 kudos
  • 1208 Views
  • 1 replies
  • 1 kudos
Latest Reply
youssefmrini
Databricks Employee
  • 1 kudos

You can ensure there is always an active run of your Databricks job with the new continuous trigger type. https://docs.databricks.com/workflows/jobs/jobs.html#continuous-jobs

  • 1 kudos
Vsleg
by Contributor
  • 2516 Views
  • 2 replies
  • 1 kudos

Resolved! Deploying Databricks Workflows and Delta Live Table pipelines across workspaces

Hello,I was wondering if there is a way to deploy Databricks Workflows and Delta Live Table pipelines across Workspaces (DEV/UAT/PROD) using Azure DevOps.

  • 2516 Views
  • 2 replies
  • 1 kudos
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 1 kudos

Yes, for sure, using Rest API Calls to https://docs.databricks.com/workflows/delta-live-tables/delta-live-tables-api-guide.htmlYou can create DLT manually from GUI and take JSON representative of it, tweak it (so it uses your env variables, for examp...

  • 1 kudos
1 More Replies
User16752245636
by New Contributor II
  • 692 Views
  • 0 replies
  • 1 kudos

Sergio Copy Data World tour Orchestration_v02.pptx (4)

Databricks Workflows: Reliable orchestration for data, analytics, and AIHello, I am a Solutions Architect at Databricks; and I recently presented at the Data and AI World Tour in London Databricks Workflows. You can see some of the presented slides a...

  • 692 Views
  • 0 replies
  • 1 kudos
Serhii
by Contributor
  • 1891 Views
  • 0 replies
  • 0 kudos

Horovod Databricks Job - custom module not found error

We have used the following example to successfully create a distributed deep learning training notebook https://www.databricks.com/blog/2022/09/07/accelerating-your-deep-learning-pytorch-lightning-databricks.html that works as expected.We now want to...

  • 1891 Views
  • 0 replies
  • 0 kudos
User16835756816
by Valued Contributor
  • 4847 Views
  • 1 replies
  • 8 kudos

Announcing: Workflows!

Databricks is excited to announce the general availability of Databricks Workflows to you, our community. Databricks Workflows is the fully managed lakehouse orchestration service for all your teams to build reliable data, analytics, and AI workflow...

  • 4847 Views
  • 1 replies
  • 8 kudos
Latest Reply
PawanShukla
New Contributor III
  • 8 kudos

I am trying to run the Workflow Pipeline with smaple code shared in getting start.. and getting the below error :DataPlaneException: Failed to start the DLT service on cluster 0526-084319-7hucy1np. Please check the stack trace below or driver logs fo...

  • 8 kudos
Labels