Announcing: Workflows!

[User16835756816]

🎉 Databricks is excited to announce the general availability of Databricks Workflows to you, our community. 🎉

Databricks Workflows is the fully managed lakehouse orchestration service for all your teams to build reliable data, analytics, and AI workflows on any cloud.

Orchestrate any combination of notebooks, SQL, Spark, ML models and dbt as a Jobs workflow, including calls to other systems. Build ETL pipelines that are automatically managed, including ingestion and lineage, using Delta Live Tables. Databricks Workflows is available on GCP, AWS and Azure, giving you full flexibility and cloud independence. 

All of you, our community members, can help power your business with fresh, reliable data and insights. 

Getting started is easy

1.) Simply log in to your workspace

2.) or sign-up for a trial!

Learn it

To learn more, check out these resources created for Workflows

• watch this Workflows demo
• read the blog for details on a 7-task workflow 
• check out our Workflows documentation
• feel free to contact us.

Join us live!

If you’re interested in learning about Workflows live from Databricks experts, watch our session at the Data & AI Summit this year.

Start the conversation! Drop your questions, feedback and tips below once you get going with Workflows! 👇

[PawanShukla]

I am trying to run the Workflow Pipeline with smaple code shared in getting start.. and getting the below error :

DataPlaneException: Failed to start the DLT service on cluster 0526-084319-7hucy1np. Please check the stack trace below or driver logs for more details.

java.util.ServiceConfigurationError: org.apache.spark.sql.sources.DataSourceRegister: Provider org.apache.spark.sql.eventhubs.EventHubsSourceProvider could not be instantiated

at java.util.ServiceLoader.fail(ServiceLoader.java:232)

[Kaniz_Fatma]

Hi @Pawan Shukla​, Can you share the sample code where you've encountered the error?

[Kaniz_Fatma]

Hi @Pawan Shukla​, , This is an issue with how your Databricks cluster is configured. Please detach your cluster and restart your cluster. Thanks.

[Kaniz_Fatma]

Hi @Pawan Shukla​, We haven't heard from you since my last response, and I was checking back to see if my suggestions helped you. Or else, If you have any solution, please share it with the community as it can be helpful to others.

Also, please don't forget to click on the "Select As Best" button whenever the information provided helps resolve your question.