topic Re: Use Case Sharing Sweepstakes ! Data + AI Summit is in full swing and we know you are just as excited as we are to learn about the new and exciting th... in Data Engineering

Use Case Sharing Sweepstakes ! Data + AI Summit is in full swing and we know you are just as excited as we are to learn about the new and exciting th...

Anonymous — Mon, 27 Jun 2022 17:56:03 GMT

Use Case Sharing Sweepstakes !

Data + AI Summit is in full swing and we know you are just as excited as we are to learn about the new and exciting things happening at Databricks. From notebooks to the Lakehouse, we know some of these new features will be game-changing for a lot of you and we want to hear about it!

In this thread we will be collecting the use cases you want to share about the new product releases and/or features you are most excited to use. The Community Member that shares the most use cases below will win an iPad!

Contest dates: Tuesday - Fri | Jun 28 - July 17

Use this template in your reply

Copy the following template for you contest entry:

Name of the new feature (give examples here)

Describe your environment (optional, but helpful)

Share 3-9 sentences describing your use case and how this new feature will help provide a new solution

What does this new feature helps you achieve that was either not possible before or required massive efforts
How can this feature affect your team productivity
What business goal or use-case is achievable relatively easily using the feature

How to win

Three participants will be chosen at random (per the Sweepstakes rules) as our Grand Prize winners and will receive an iPad Air. 🎉

Click here for official contest rules

Re: Use Case Sharing Sweepstakes ! Data + AI Summit is in full swing and we know you are just as excited as we are to learn about the new and exciting th...

Hubert-Dudek — Wed, 29 Jun 2022 14:27:32 GMT

Libraries in the Pool.

Currently, we specify libraries on the cluster level. Cluster is installing Libraries whenever cluster is deploying from the Pool. Adding install libraries on the Pool will be helpful as it will limit cluster deployment time from the Pool.

Re: Use Case Sharing Sweepstakes ! Data + AI Summit is in full swing and we know you are just as excited as we are to learn about the new and exciting th...

Hubert-Dudek — Wed, 29 Jun 2022 14:27:45 GMT

Trigger job based on the event - when last modified

Event-driven job triggering. A job is triggered when the insert/update/delete option updates the table. Such an option would be excellent as an optional parameter.

Re: Use Case Sharing Sweepstakes ! Data + AI Summit is in full swing and we know you are just as excited as we are to learn about the new and exciting th...

Hubert-Dudek — Wed, 29 Jun 2022 14:27:56 GMT

Databricks integration with Logic Apps / Power Automate (job triggering etc.)

Azure logic apps, Power Automate, zappier, etc., are prevalent nowadays. All of them are missing integration with Databricks. There should be at least a possibility to automate trigger specified job/notebook (like in Azure Data Factory)

Re: Use Case Sharing Sweepstakes ! Data + AI Summit is in full swing and we know you are just as excited as we are to learn about the new and exciting th...

Hubert-Dudek — Wed, 29 Jun 2022 14:28:06 GMT

Cron

Cron syntax in workflows is nice but is missing some functionality. Like for example, to run the job every 40 second (for example 12:40:00, 12:40:40, 12:41:20)

Re: Use Case Sharing Sweepstakes ! Data + AI Summit is in full swing and we know you are just as excited as we are to learn about the new and exciting th...

Hubert-Dudek — Wed, 29 Jun 2022 14:28:16 GMT

Workflow continuously

Workflows should have the option to work continuously. When workflow execution finishes (or fails), start it immediately again. Currently, I need to set a job every minute and have most of the runs skipped.

Re: Use Case Sharing Sweepstakes ! Data + AI Summit is in full swing and we know you are just as excited as we are to learn about the new and exciting th...

Hubert-Dudek — Wed, 29 Jun 2022 14:28:26 GMT

SQL in standard version (can be charged as in premium)

Many people register for standard databricks workplace and never see Delta live tables or SQL. I think databricks miss revenue this way. Those options should be available in standard but should be charged as in premium.

Re: Use Case Sharing Sweepstakes ! Data + AI Summit is in full swing and we know you are just as excited as we are to learn about the new and exciting th...

Hubert-Dudek — Wed, 29 Jun 2022 14:28:35 GMT

Repo/Workplace Browser

Using slide Repo/workspace can be frustrating. It could be nice to have the option not to have it as a sidebar but as a full screen.

Re: Use Case Sharing Sweepstakes ! Data + AI Summit is in full swing and we know you are just as excited as we are to learn about the new and exciting th...

Hubert-Dudek — Wed, 29 Jun 2022 14:28:43 GMT

Multi tabs - notebooks

Like in the SQL experience, having multiple tabs with open notebooks could be nice.

Re: Use Case Sharing Sweepstakes ! Data + AI Summit is in full swing and we know you are just as excited as we are to learn about the new and exciting th...

Hubert-Dudek — Wed, 29 Jun 2022 14:28:51 GMT

Log cleaner etc. via API

To clean the workspace, we need to go through the Admin console every time. So it would be nice to have everything in the admin console available through REST API.

Re: Use Case Sharing Sweepstakes ! Data + AI Summit is in full swing and we know you are just as excited as we are to learn about the new and exciting th...

Hubert-Dudek — Wed, 29 Jun 2022 14:29:00 GMT

Send SQL queries to SQL endpoint via API

Yes, it was on my list for a long time, but I've just heard that SQL API/SDK is coming.

Re: Use Case Sharing Sweepstakes ! Data + AI Summit is in full swing and we know you are just as excited as we are to learn about the new and exciting th...

Hubert-Dudek — Wed, 29 Jun 2022 15:00:08 GMT

Delta Live Tables from the notebook

It would be nice to run live delta tables from the notebook, too (so the dlt library will be available there). It could at least not throw the error (ignore that part)

Re: Use Case Sharing Sweepstakes ! Data + AI Summit is in full swing and we know you are just as excited as we are to learn about the new and exciting th...

Hubert-Dudek — Wed, 29 Jun 2022 15:01:53 GMT

Delta Live Tables %RUN

Delta lives tables should support %RUN as it creates spaghetti code. Or should be another option to import functions from another notebook (currently, you can import only functions from files in the repo, not from the notebook).

Re: Use Case Sharing Sweepstakes ! Data + AI Summit is in full swing and we know you are just as excited as we are to learn about the new and exciting th...

ranged_coop — Wed, 06 Jul 2022 07:00:11 GMT

Had entered below items thinking the contest was for future use cases..

posting them anyways - hoping it would be useful someday...

Here is a list of items we wish were available....

Workflow Failure Options:

Currently we have the option to set dependency based on job success.

Would be nice to have the option to trigger a job in case a job fails.

Conditional Workflow Options:

Similar to a scheduler - would be nice to have the option to trigger a job based on a variable or parameter across jobs. Say file name from job 1 passed to the job 2 or trigger different set of jobs based on the status code obtained in first job.

Custom E-Mail Options with Attachments:

Send a customized e-mail instead of the standard failure/success mail along with attachments.

Workflows - Filter and Sort from the UI:

Easier way to filter out currently running jobs, sort based on running time and not the last execution status.

Easier Way to bring data into Lake:

Easier way to bring data into the lake, like a ready made SFTP option that would directly import data into the datalake to be acted upon further by Databricks.

Re: Use Case Sharing Sweepstakes ! Data + AI Summit is in full swing and we know you are just as excited as we are to learn about the new and exciting th...

AmanSehgal — Wed, 13 Jul 2022 05:49:18 GMT

Spark Connect

Running spark from edge devices is going to be a game changer. As AI/ML capabilities are growing in edge devices like cameras, industrial sensors, wearable devices in medical industries - having the potential to run Spark from them is going to be a lot more beneficial for scenarios where precise predictions can make a difference.

A lot more open source devices can leverage this capability and go beyond their available resources.

Looking forward for documentation on Spark Connect and its library.

Re: Use Case Sharing Sweepstakes ! Data + AI Summit is in full swing and we know you are just as excited as we are to learn about the new and exciting th...

AmanSehgal — Wed, 13 Jul 2022 13:08:31 GMT

Serverless SQL Warehouses

This is really great work by Databricks. 10 seconds to launch a cluster that can start executing queries right away. It'll help our analytics team to start crunching data for any use case or report they want to generate. It's faster, it's quicker and accessing data is much more easier now.

Re: Use Case Sharing Sweepstakes ! Data + AI Summit is in full swing and we know you are just as excited as we are to learn about the new and exciting th...

AmanSehgal — Wed, 13 Jul 2022 13:16:11 GMT

Unity Catalog

Another great tool from Databricks. I'd been waiting for this to be launched since the announcement.

Unity has helped in performing data discovery and providing access control with a UI. It helps admin to monitor data access by different users - which in turn serves as audit report as well

Re: Use Case Sharing Sweepstakes ! Data + AI Summit is in full swing and we know you are just as excited as we are to learn about the new and exciting th...

AmanSehgal — Thu, 14 Jul 2022 09:30:39 GMT

Interested to see Folder structure for Databricks SQL

The way notebooks are managed per account basis within folders, it will be really beneficial to have DB SQLs organized within folders and not be presented in a flat hierarchy.

This will help in searching, organizing and maintaining DB SQL notebooks. Currently, the only way to search a notebook is via tags and that's not very helpful

Re: Use Case Sharing Sweepstakes ! Data + AI Summit is in full swing and we know you are just as excited as we are to learn about the new and exciting th...

AmanSehgal — Sat, 16 Jul 2022 02:45:13 GMT

SQL REST API for Python, Go clients

Definitely going to assist well in integrating delta lake with web applications used for analytics. If integrated with SQL serverless, then analytics applications can directly run over data lake using REST APIs. Moreover, now Go and nodejs developers can onboard on databricks platform.

Re: Use Case Sharing Sweepstakes ! Data + AI Summit is in full swing and we know you are just as excited as we are to learn about the new and exciting th...

AmanSehgal — Sat, 16 Jul 2022 02:48:44 GMT

Delta Sharing

There have been several instances, where clients want to share data within departments or to external parties - and usually this is done via CSV files, JSON or XML files. It takes a lot of effort to clean data, validate it, match the schema and then load into target database.

With delta sharing this is going to be lot more easier and streamlined especially now with support for CDF and compatibility with node, powerBi, Go and Airflow.