โ06-27-2022 10:56 AM
Use Case Sharing Sweepstakes !
Data + AI Summit is in full swing and we know you are just as excited as we are to learn about the new and exciting things happening at Databricks. From notebooks to the Lakehouse, we know some of these new features will be game-changing for a lot of you and we want to hear about it!
In this thread we will be collecting the use cases you want to share about the new product releases and/or features you are most excited to use. The Community Member that shares the most use cases below will win an iPad!
Contest dates: Tuesday - Fri | Jun 28 - July 17
Use this template in your reply
Copy the following template for you contest entry:
Name of the new feature (give examples here)
Describe your environment (optional, but helpful)
Share 3-9 sentences describing your use case and how this new feature will help provide a new solution
How to win
Three participants will be chosen at random (per the Sweepstakes rules) as our Grand Prize winners and will receive an iPad Air. ๐
โ06-29-2022 07:27 AM
Libraries in the Pool.
Currently, we specify libraries on the cluster level. Cluster is installing Libraries whenever cluster is deploying from the Pool. Adding install libraries on the Pool will be helpful as it will limit cluster deployment time from the Pool.
โ06-29-2022 07:27 AM
Trigger job based on the event - when last modified
Event-driven job triggering. A job is triggered when the insert/update/delete option updates the table. Such an option would be excellent as an optional parameter.
โ06-29-2022 07:27 AM
Databricks integration with Logic Apps / Power Automate (job triggering etc.)
Azure logic apps, Power Automate, zappier, etc., are prevalent nowadays. All of them are missing integration with Databricks. There should be at least a possibility to automate trigger specified job/notebook (like in Azure Data Factory)
โ06-29-2022 07:28 AM
Cron
Cron syntax in workflows is nice but is missing some functionality. Like for example, to run the job every 40 second (for example 12:40:00, 12:40:40, 12:41:20)
โ06-29-2022 07:28 AM
Workflow continuously
Workflows should have the option to work continuously. When workflow execution finishes (or fails), start it immediately again. Currently, I need to set a job every minute and have most of the runs skipped.
โ06-29-2022 07:28 AM
SQL in standard version (can be charged as in premium)
Many people register for standard databricks workplace and never see Delta live tables or SQL. I think databricks miss revenue this way. Those options should be available in standard but should be charged as in premium.
โ06-29-2022 07:28 AM
Repo/Workplace Browser
Using slide Repo/workspace can be frustrating. It could be nice to have the option not to have it as a sidebar but as a full screen.
โ06-29-2022 07:28 AM
Multi tabs - notebooks
Like in the SQL experience, having multiple tabs with open notebooks could be nice.
โ06-29-2022 07:28 AM
Log cleaner etc. via API
To clean the workspace, we need to go through the Admin console every time. So it would be nice to have everything in the admin console available through REST API.
โ06-29-2022 07:29 AM
Send SQL queries to SQL endpoint via API
Yes, it was on my list for a long time, but I've just heard that SQL API/SDK is coming.
โ06-29-2022 08:00 AM
Delta Live Tables from the notebook
It would be nice to run live delta tables from the notebook, too (so the dlt library will be available there). It could at least not throw the error (ignore that part)
โ06-29-2022 08:01 AM
Delta Live Tables %RUN
Delta lives tables should support %RUN as it creates spaghetti code. Or should be another option to import functions from another notebook (currently, you can import only functions from files in the repo, not from the notebook).
โ07-06-2022 12:00 AM
Had entered below items thinking the contest was for future use cases..
posting them anyways - hoping it would be useful someday...
Here is a list of items we wish were available....
Workflow Failure Options:
Currently we have the option to set dependency based on job success.
Would be nice to have the option to trigger a job in case a job fails.
Conditional Workflow Options:
Similar to a scheduler - would be nice to have the option to trigger a job based on a variable or parameter across jobs. Say file name from job 1 passed to the job 2 or trigger different set of jobs based on the status code obtained in first job.
Custom E-Mail Options with Attachments:
Send a customized e-mail instead of the standard failure/success mail along with attachments.
Workflows - Filter and Sort from the UI:
Easier way to filter out currently running jobs, sort based on running time and not the last execution status.
Easier Way to bring data into Lake:
Easier way to bring data into the lake, like a ready made SFTP option that would directly import data into the datalake to be acted upon further by Databricks.
โ07-12-2022 10:49 PM
Spark Connect
Running spark from edge devices is going to be a game changer. As AI/ML capabilities are growing in edge devices like cameras, industrial sensors, wearable devices in medical industries - having the potential to run Spark from them is going to be a lot more beneficial for scenarios where precise predictions can make a difference.
A lot more open source devices can leverage this capability and go beyond their available resources.
Looking forward for documentation on Spark Connect and its library.
Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโt want to miss the chance to attend and share knowledge.
If there isnโt a group near you, start one and help create a community that brings people together.
Request a New Group