cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

Start a conversation
akihiko
by • New Contributor III
  • 3299 Views
  • 4 replies
  • 1 kudos

Resolved! Attach notebook to cluster via REST API

Is it possible to attach a notebook to cluster and run it via the REST API?The closest approach I have found is to run a notebook, export the results (HTML!) and import it into the workspace again, but this does not allow us to retain the original ex...

Data Engineering
Reply
  • 3299 Views
  • 4 replies
  • 1 kudos
View Replies
Latest Reply
baert23
New Contributor II
  • 1 kudos

I'm looking for a way to programmatically copy a notebook in Databricks using the workspace/export and workspace/import APIs. Once the notebook is copied, I want to automatically attach it to a specific cluster using its cluster ID. The challenge is ...

  • 1 kudos
3 More Replies
Rishabh-Pandey
by • Esteemed Contributor
  • 543 Views
  • 2 replies
  • 1 kudos

Resolved! The Latest Improvements to Databricks Workflows

What's new in Workflows?  @Sujitha @Retired_mod

Data Engineering
Reply
  • 543 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
Rishabh-Pandey
Esteemed Contributor
  • 1 kudos

@Sujitha I am happy to see workflows maturing day by day; this is going to be a game changer for the market. I am also very excited about the upcoming feature, Lakeflow.

  • 1 kudos
1 More Replies
VIRALKUMAR
by • Contributor II
  • 1440 Views
  • 2 replies
  • 0 kudos

How to Determine the Cost for Each Query Run Against SQL Warehouse Serverless?

Hello Everyone.First of all, I would like to thank you to databricks to enable system tables for customers. It does help a lot. I am working on cost optimization topic. Particularly sql warehouse serverless. I am not sure all of you have tried system...

Data Engineering
Reply
  • 1440 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
katefray
Databricks Employee
  • 0 kudos

Hey VIRALKUMAR, I recommend using the billing usage system table to find total DBUs by SKU (SQL) and the pricing system table to find the appropriate price. You can use the sample queries in those pages to get started. Hope that's helpful!

  • 0 kudos
1 More Replies
CURIOUS_DE
by • New Contributor III
  • 288 Views
  • 0 replies
  • 1 kudos

A Surprise Findings in Delta Live Table

While DLT has some powerful features, I found myself doing a double-take when I realized it doesn’t natively support hard deletes. Instead, it leans on a delete flag identifier to manage these in the source table. A bit surprising for a tool of its c...

Data Engineering
Reply
  • 288 Views
  • 0 replies
  • 1 kudos
RobsonNLPT
by • Contributor II
  • 364 Views
  • 1 replies
  • 0 kudos

Compute Serverless Databricks Runtime

Hi all.2 questions about compute serverless1) How to check or query the runtime version used by Serverless compute/workflow? This is important as I use some features supported by specific runtime or higher2) Can you confirm if spark.conf.set(......) ...

Data Engineering
Reply
  • 364 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Witold
Contributor III
  • 0 kudos

Sure: https://learn.microsoft.com/en-us/azure/databricks/sql/language-manual/functions/current_version and https://learn.microsoft.com/en-us/azure/databricks/sql/language-manual/functions/versionOnly some properties are allowed to be changed 

  • 0 kudos
DaPo
by • New Contributor II
  • 583 Views
  • 0 replies
  • 0 kudos

DLT: Create Empty Table If Autloader Fails

Hi all,I am quite new to databricks. Overall I enjoyed the experience so far, but now ran into a problem, I was not able to find an acceptable solution.Here is my setup: I have a bunch of s3 buckets, and need to put the data into databricks, preferab...

Data Engineering
autoloader
Delta Live Table
dlt
spark
Reply
  • 583 Views
  • 0 replies
  • 0 kudos
drag7ter
by • New Contributor III
  • 379 Views
  • 0 replies
  • 0 kudos

foreachBatch doesn't work in structured streaming

I' m trying to print out number of rows in the batch, but seems it doesn't work properly. I have 1 node compute optimized cluster and run in notebook this code:# Logging the row count using a streaming-friendly approach def log_row_count(batch_df, ba...

Capture.PNG
Data Engineering
Reply
  • 379 Views
  • 0 replies
  • 0 kudos
kumarPatra_07
by • New Contributor
  • 270 Views
  • 0 replies
  • 0 kudos

getting short of error while mount to storage account.

while mount to the storage account using this below code dbutils.fs.mount(  source=f"wasbs://{cointainer_name}@{storage_name}.blob.core.windows.net",  mount_point=f"/mnt/{cointainer_name}",  extra_configs={f"fs.azure.account.key.{storage_name}.blob.c...

Data Engineering
Reply
  • 270 Views
  • 0 replies
  • 0 kudos
jamson
by • New Contributor
  • 1853 Views
  • 1 replies
  • 0 kudos

What are the best practices for optimizing Power BI reports and dashboards for performance in the PL

I’m studying for the PL-300 exam and would love some advice on how to optimize Power BI reports and dashboards for better performance. Specifically, I’m interested in:Techniques for improving report load times and responsiveness.Best practices for ma...

Data Engineering
Reply
  • 1853 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
szymon_dybczak
Contributor III
  • 0 kudos

Hi @jamson ,Here's a good article that answers your question. I think author did a pretty good job - many of his advice I apply on everyday job.How I Tuned Databricks Query Performance from Power BI Desktop: A Personal Journey. | by Brahmareddy, The ...

  • 0 kudos
yagmur
by • New Contributor II
  • 778 Views
  • 2 replies
  • 1 kudos

writing to blob storage from databricks (parquet format)

Hi, I am supposed to create transformation notebook. But i am having trouble when i am trying to save the transformed file into blob storage. I didn't use any layer, just the layer which i am performing transformation in. if i use wasbs i receive dif...

Data Engineering
Reply
  • 778 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
szymon_dybczak
Contributor III
  • 1 kudos

Hi @yagmur ,Did you assigned required permission to service principal on storage account?And make sure you're configuring connection to storage account in proper way. You should have something similiar to the code below: configs = { "fs.a...

  • 1 kudos
1 More Replies
A4Ashish
by • New Contributor II
  • 610 Views
  • 2 replies
  • 0 kudos

Paramter Type while Executing Rest API of Execute a SQL statement POST /api/2.0/sql/statements/

Hi,So previously I was using Old Preview Databricks API version. Now I have switched to v1 of the API which usage Execute a SQL statement POST /api/2.0/sql/statements/I wanted to know how to pass the parameter class when my value is list of string fo...

Data Engineering
Reply
  • 610 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
A4Ashish
New Contributor II
  • 0 kudos

Hi @szymon_dybczak Any idea how to better handle IN operator with a list in new API version

  • 0 kudos
1 More Replies
User16826992666
by • Valued Contributor
  • 2200 Views
  • 2 replies
  • 1 kudos

Resolved! If someone saves a flat file in a Databricks notebook without specifying a location, where does it go?

I ran the code block below and now I can't find the file. Where would this get saved since no location was specified?

Cmd
Data Engineering
Reply
  • 2200 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
jecastelo
New Contributor II
  • 1 kudos

/Workspace/Users/your.email@companyname.com/myfile.txt

  • 1 kudos
1 More Replies
Dave_Nithio
by • Contributor
  • 9264 Views
  • 6 replies
  • 7 kudos

Resolved! Delta Live Table Pipeline with Multiple Notebooks

I have two notebooks created for my Delta Live Table pipeline. The first is a utils notebook with functions I will be reusing for other pipelines. The second contains my actual creation of the delta live tables. I added both notebooks to the pipeline...

image.png
Data Engineering
Reply
  • 9264 Views
  • 6 replies
  • 7 kudos
View Replies
Latest Reply
JackyL
New Contributor II
  • 7 kudos

Hi Dave,You can solve this by putting your utils into a python file and referencing your .py file in the DLT notebook. I provided a template for the python file below:STEP 1: #import functions from pyspark.sql import SparkSession import IPython dbut...

  • 7 kudos
5 More Replies

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels