Use Case Sharing Sweepstakes ! Data + AI Summit is in full swing and we know you are just as excited as we are to learn about the new and exciting th...

[Anonymous]

Use Case Sharing Sweepstakes !

Data + AI Summit is in full swing and we know you are just as excited as we are to learn about the new and exciting things happening at Databricks. From notebooks to the Lakehouse, we know some of these new features will be game-changing for a lot of you and we want to hear about it!

In this thread we will be collecting the use cases you want to share about the new product releases and/or features you are most excited to use. The Community Member that shares the most use cases below will win an iPad! 

Contest dates: Tuesday - Fri | Jun 28 - July 17 

Use this template in your reply 

Copy the following template for you contest entry:

Name of the new feature (give examples here)

Describe your environment (optional, but helpful)

Share 3-9 sentences describing your use case and how this new feature will help provide a new solution

• What does this new feature helps you achieve that was either not possible before or required massive efforts
• How can this feature affect your team productivity
• What business goal or use-case is achievable relatively easily using the feature

How to win

Three participants will be chosen at random (per the Sweepstakes rules) as our Grand Prize winners and will receive an iPad Air. 🎉

Click here for official contest rules

[Hubert-Dudek]

Libraries in the Pool.

Currently, we specify libraries on the cluster level. Cluster is installing Libraries whenever cluster is deploying from the Pool. Adding install libraries on the Pool will be helpful as it will limit cluster deployment time from the Pool.

[Hubert-Dudek]

Trigger job based on the event - when last modified

Event-driven job triggering. A job is triggered when the insert/update/delete option updates the table. Such an option would be excellent as an optional parameter.

[Hubert-Dudek]

Databricks integration with Logic Apps / Power Automate (job triggering etc.)

Azure logic apps, Power Automate, zappier, etc., are prevalent nowadays. All of them are missing integration with Databricks. There should be at least a possibility to automate trigger specified job/notebook (like in Azure Data Factory)

[Hubert-Dudek]

Cron

Cron syntax in workflows is nice but is missing some functionality. Like for example, to run the job every 40 second (for example 12:40:00, 12:40:40, 12:41:20)

[Hubert-Dudek]

Workflow continuously

Workflows should have the option to work continuously. When workflow execution finishes (or fails), start it immediately again. Currently, I need to set a job every minute and have most of the runs skipped.

[Hubert-Dudek]

SQL in standard version (can be charged as in premium)

Many people register for standard databricks workplace and never see Delta live tables or SQL. I think databricks miss revenue this way. Those options should be available in standard but should be charged as in premium.

[Hubert-Dudek]

Repo/Workplace Browser

Using slide Repo/workspace can be frustrating. It could be nice to have the option not to have it as a sidebar but as a full screen.

[Hubert-Dudek]

Multi tabs - notebooks

Like in the SQL experience, having multiple tabs with open notebooks could be nice.

[Hubert-Dudek]

Log cleaner etc. via API

To clean the workspace, we need to go through the Admin console every time. So it would be nice to have everything in the admin console available through REST API.

[Hubert-Dudek]

Send SQL queries to SQL endpoint via API

Yes, it was on my list for a long time, but I've just heard that SQL API/SDK is coming.

[Hubert-Dudek]

Delta Live Tables from the notebook

It would be nice to run live delta tables from the notebook, too (so the dlt library will be available there). It could at least not throw the error (ignore that part)

[Hubert-Dudek]

Delta Live Tables %RUN

Delta lives tables should support %RUN as it creates spaghetti code. Or should be another option to import functions from another notebook (currently, you can import only functions from files in the repo, not from the notebook).

[ranged_coop]

Had entered below items thinking the contest was for future use cases..

posting them anyways - hoping it would be useful someday...

Here is a list of items we wish were available....

Workflow Failure Options:

Currently we have the option to set dependency based on job success.

Would be nice to have the option to trigger a job in case a job fails.

Conditional Workflow Options:

Similar to a scheduler - would be nice to have the option to trigger a job based on a variable or parameter across jobs. Say file name from job 1 passed to the job 2 or trigger different set of jobs based on the status code obtained in first job.

Custom E-Mail Options with Attachments:

Send a customized e-mail instead of the standard failure/success mail along with attachments.

Workflows - Filter and Sort from the UI:

Easier way to filter out currently running jobs, sort based on running time and not the last execution status.

Easier Way to bring data into Lake:

Easier way to bring data into the lake, like a ready made SFTP option that would directly import data into the datalake to be acted upon further by Databricks.

[AmanSehgal]

Spark Connect

Running spark from edge devices is going to be a game changer. As AI/ML capabilities are growing in edge devices like cameras, industrial sensors, wearable devices in medical industries - having the potential to run Spark from them is going to be a lot more beneficial for scenarios where precise predictions can make a difference.

 A lot more open source devices can leverage this capability and go beyond their available resources.

Looking forward for documentation on Spark Connect and its library.