cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
Showing results for 
Search instead for 
Did you mean: 
Data Intelligence for Data Engineers

Join us to find out how a platform built on lakehouse architecture and enhanced with built-in data intelligence automates many of the tasks that bog down engineers. You’ll discover how the Databricks Data Intelligence Platform helps you build secure...

  • 36 Views
  • 0 replies
  • 0 kudos
yesterday

Community Activity

KalyaniJaya
by > Visitor
  • 2 Views
  • 0 replies
  • 0 kudos

'dbutils.jobs.taskValues.get' taking debug value in workflow, instead of actual value being set

Hi,I am trying to pass and set values from one wheel into another wheel in databricks workflow.I have used 'dbutils.jobs.taskValues.get' and 'dbutils.jobs.taskValues.set'I have used 'dbutils.jobs.taskValues.get' in second task and made sure to keep d...

  • 2 Views
  • 0 replies
  • 0 kudos
pranitha
by > New Contributor
  • 45 Views
  • 2 replies
  • 1 kudos
  • 45 Views
  • 2 replies
  • 1 kudos
Latest Reply
Walter_C
Databricks Employee
  • 1 kudos

Azure provides an Activity Log that records all write operations (PUT, POST, DELETE) for your storage account. This log includes information about who initiated the operation and when it occurred.

  • 1 kudos
1 More Replies
MatthewMills
by > New Contributor III
  • 132 Views
  • 1 replies
  • 4 kudos

DLT Apply Changes Tables corrupt

Got a weird DLT error.Test harness using the new(ish) 'Apply Changes from Snapshot' Functionality and DLT Serverless (Current Channel). Azure Aus East Region.Has been working for several months without issue - but within the last week these DLT table...

Data Engineering
Apply Changes From Snapshot
dlt
  • 132 Views
  • 1 replies
  • 4 kudos
Latest Reply
Pelle123
New Contributor II
  • 4 kudos

I'm experiencing the same issue.

  • 4 kudos
JulianKrüger
by > New Contributor
  • 18 Views
  • 0 replies
  • 0 kudos

Limited concurrent running DLT's within a pipeline

Hi Champions!We are currently experiencing a few unexplainable limitations when executing pipelines with > 50 DLT tables. It looks like, that there is some calculation in the background in place, to determine the maximum number of concurrent running ...

JulianKrger_0-1737625978665.png JulianKrger_1-1737626627178.png JulianKrger_2-1737626698802.png
Data Engineering
dlt
pipeline
  • 18 Views
  • 0 replies
  • 0 kudos
ledsouza
by > New Contributor
  • 603 Views
  • 2 replies
  • 0 kudos

Community Edition workspace not found

Suddenly got logout from my account in the Community Edition. When I tried to login again, I received this error message: "We were not able to find a Community Edition workspace with this email. Please login to accounts.cloud.databricks.com to find t...

  • 603 Views
  • 2 replies
  • 0 kudos
Latest Reply
Advika
Databricks Employee
  • 0 kudos

Hello @ledsouza and @vinitptdr! You can find helpful resources for Databricks Community Edition here. If the available resource doesn’t resolve your concern, please submit a ticket with the Databricks Support team for further assistance. Thank you.

  • 0 kudos
1 More Replies
ankitmit
by > New Contributor II
  • 36 Views
  • 0 replies
  • 0 kudos

How to specify path while creating tables using DLT

Hi All,I am trying to create table using DLT and would like to specify the path where all the files should reside.I am trying something like this:dlt.create_streaming_table( name="test", schema="""product_id STRING NOT NULL PRIMARY KEY, ...

Data Engineering
Databricks
dlt
Unity Catalog
  • 36 Views
  • 0 replies
  • 0 kudos
jonxu
by > New Contributor III
  • 67 Views
  • 3 replies
  • 0 kudos

minor bug in Lab 2 of ML Academy

# Remove listings with MentHlth > -40 should be  # Remove listings with MentHlth < -40

  • 67 Views
  • 3 replies
  • 0 kudos
Latest Reply
Advika
Databricks Employee
  • 0 kudos

Hello @jonxu!Could you please provide more details about the course and the specific Lab you are referring to?

  • 0 kudos
2 More Replies
idanyow
by > New Contributor II
  • 269 Views
  • 8 replies
  • 1 kudos

01_demo_setup error

HelloI was following "Demo: Creating and Working with a Delta Table"while I have a community edition user.The first command in the Notebook is: %run ./setup/01_demo_setup But I got the following error:Notebook not found: Users/<my-email-was-here..>/s...

  • 269 Views
  • 8 replies
  • 1 kudos
Latest Reply
idanyow
New Contributor II
  • 1 kudos

Still facing it myself. Any solution ?

  • 1 kudos
7 More Replies
norbitek
by > New Contributor II
  • 15 Views
  • 0 replies
  • 0 kudos

Identity column and impact on performance

Hi,I want to define identity column in the Delta table.Based on documentation:"Declaring an identity column on a Delta table disables concurrent transactions. Only use identity columns in use cases where concurrent writes to the target table are not ...

  • 15 Views
  • 0 replies
  • 0 kudos
Nate-Haines
by > New Contributor
  • 96 Views
  • 3 replies
  • 4 kudos

Drive memory utilization cleanup

Issue Summary:When running multiple jobs on the same compute cluster, over time, I see an increase in memory utilization that is seemingly never fully released, even when jobs finish. This eventually leads to some jobs stalling out as memory hits the...

  • 96 Views
  • 3 replies
  • 4 kudos
Latest Reply
KyleGrymonpre
  • 4 kudos

I'm encountering something similar. Immediately upon starting a cluster and triggering a job run, my memory usage jumps from 0 to about 20GB used and 15GB cached (see the attached screenshot). The data I am working with should be very small (less tha...

  • 4 kudos
2 More Replies
Avinash_Narala
by > Valued Contributor
  • 6 Views
  • 0 replies
  • 0 kudos

Python udf to pyspark udf conversion

Hi,I want to convert my python udf to pyspark udf, is there any guide/article on suggesting the best practices and avoid miscalculations if any 

  • 6 Views
  • 0 replies
  • 0 kudos
pradeepvatsvk
by > New Contributor
  • 19 Views
  • 2 replies
  • 2 kudos

Working with Pandas through Abfss

Hi, I am unable to read and write pandas datframe through abfss protocol , Is there a work around for this , I do not want to store my files in dbfs

  • 19 Views
  • 2 replies
  • 2 kudos
Latest Reply
Avinash_Narala
Valued Contributor
  • 2 kudos

you can use volumes and mount the abfss location to unity catalog and can access the files present in azure in databricks. Regards,Avinash N

  • 2 kudos
1 More Replies
pwtnew32
by > New Contributor II
  • 57 Views
  • 3 replies
  • 3 kudos

Lakehouse Federation

I use Lakehouse Federation to connect hive metastore (local VM) with metastore database type mysqlIt's can see database and table in hive but when I query data session continue running without failed or success.Do I have to migrate data to ADLs which...

  • 57 Views
  • 3 replies
  • 3 kudos
Latest Reply
Avinash_Narala
Valued Contributor
  • 3 kudos

As for Lakehouse Federation, only some sources are supported as of now. In that, connecting to HiveMetastore data which is in local VM is not supported, so you can migrate that data to ADLS and mount that as external location in unity catalog and que...

  • 3 kudos
2 More Replies
yvishal519
by > Contributor
  • 22 Views
  • 1 replies
  • 0 kudos

Identifying Full Refresh vs. Incremental Runs in Delta Live Tables

Hello Community,I am working with a Delta Live Tables (DLT) pipeline that primarily operates in incremental mode. However, there are specific scenarios where I need to perform a full refresh of the pipeline. I am looking for an efficient and reliable...

  • 22 Views
  • 1 replies
  • 0 kudos
Latest Reply
daniel598lopez
  • 0 kudos

@yvishal519 wrote:Hello Community,I am working with a Delta Live Tables (DLT) pipeline that primarily operates in incremental mode. However, there are specific scenarios where I need to perform a full refresh of the pipeline. I am looking for an effi...

  • 0 kudos
ampammam
by > Visitor
  • 16 Views
  • 1 replies
  • 0 kudos

Driver log storage location

What directory would the driver log normally be stored in? Is it DBFS?

  • 16 Views
  • 1 replies
  • 0 kudos
Latest Reply
szymon_dybczak
Esteemed Contributor III
  • 0 kudos

Hi @ampammam ,The default storage location for driver logs in Databricks is on the local disk of the driver node. I think you can change this behavior going to cluster settings: Advanced Options→Logging→Destination 

  • 0 kudos

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Top Kudoed Authors
Read Databricks Data Intelligence Platform reviews on G2

Latest from our Blog

Deep Dive - Streaming Deduplication

In this article we will cover in depth about streaming deduplication using watermarking with dropDuplicates and dropDuplicatesWithinWatermark, how they are different. This blog expects you to have a g...

565Views 1kudos

Data Engineering SQL Holiday Specials

December is the most celebrated time of year in the Data Engineering calendar as we embrace the important holiday: change freeze season.  As we come back to the office to start our new projects, I wan...

3128Views 3kudos