cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

shervinmir
by New Contributor II
  • 4540 Views
  • 4 replies
  • 0 kudos

Using user-assigned managed identity inside notebook

Hi team,I am interested in using a user-assigned managed identity within my notebook. I've come across examples using system-assigned managed identities or leveraging the Access Connector for Azure Databricks via Unity Catalog. However, as I do not h...

  • 4540 Views
  • 4 replies
  • 0 kudos
Latest Reply
shervinmir
New Contributor II
  • 0 kudos

Hi team, Just wondering if anyone has any suggestions. We are still unable to use User Assigned managed identity inside the a notebook in Databricks to connect to a external Gen 2 storage 

  • 0 kudos
3 More Replies
Prashanth24
by New Contributor III
  • 9969 Views
  • 8 replies
  • 4 kudos

Resolved! Difference between Liquid clustering and Z-ordering

I am trying to understand the difference between Liquid clustering and z-ordering. As per my understanding, both stores the clustered information into ZCubes which is of size 100 GB.Liquid Clustering maintains ZCube id in transaction log so when opti...

  • 9969 Views
  • 8 replies
  • 4 kudos
Latest Reply
canadiandataguy
New Contributor III
  • 4 kudos

I have built a decision tree on how to think about it https://www.canadiandataguy.com/p/optimizing-delta-lake-tables-liquid?triedRedirect=true

  • 4 kudos
7 More Replies
alextc77
by New Contributor II
  • 2212 Views
  • 3 replies
  • 2 kudos

Resolved! Unable to Access Unity Catalog from Cluster in Azure Databricks while Serverless Works

I can't access tables and volumes from the Unity Catalog using a cluster in Azure Databricks, although it works with serverless. Why is this the case?※As for the cluster, the summary displayed the "UnityCatalog" UC badge, and the access mode (data_se...

  • 2212 Views
  • 3 replies
  • 2 kudos
Latest Reply
Brahmareddy
Esteemed Contributor
  • 2 kudos

Thanks for the updates, Alex. Good day.

  • 2 kudos
2 More Replies
j_h_robinson
by New Contributor II
  • 2672 Views
  • 0 replies
  • 0 kudos

GitHub CI/CD Best Practices

Using GitHub, what are some best-practice CI/CD approaches to use specifically with the silver and gold medallion layers? We want to create the bronze, silver, and gold layers in Databricks notebooks.Also, is using notebooks in production a "best pra...

  • 2672 Views
  • 0 replies
  • 0 kudos
SObiero
by New Contributor
  • 2702 Views
  • 0 replies
  • 0 kudos

Passing Microsoft MFA Auth from Databricks to MSSQL Managed Instance in a Databricks FastAPI App

I have a Databricks App built using FastAPI. Users access this App after authenticating with Microsoft MFA on Databricks Azure Cloud. The App connects to an MSSQL Managed Instance (MI) that also supports Microsoft MFA.I want the authenticated user's ...

  • 2702 Views
  • 0 replies
  • 0 kudos
chitrar
by New Contributor III
  • 1816 Views
  • 9 replies
  • 4 kudos

workflow/lakeflow -why does it not capture all the metadata of the jobs/tasks

Hi, I see with unity catalog we have the workflow and now the lakeflow schema.   I guess the intention is to capture audit logs of changes/ monitor runs but I wonder why we don't have all the  metadata  info on the jobs /tasks too for a given job   =...

  • 1816 Views
  • 9 replies
  • 4 kudos
Latest Reply
chitrar
New Contributor III
  • 4 kudos

@Sujitha  so, we can expect these enhancements in the "near" future ?

  • 4 kudos
8 More Replies
antoniomf
by New Contributor
  • 2792 Views
  • 0 replies
  • 0 kudos

Bug Delta Live Tables - Checkpoint

Hello, I've encountered an issue with Delta Live Table in both my Development and Production Workspaces. The data is arriving correctly in my Azure Storage Account; however, the checkpoint is being stored in the path dbfs:/. I haven't modified the St...

  • 2792 Views
  • 0 replies
  • 0 kudos
jeremy98
by Honored Contributor
  • 2753 Views
  • 1 replies
  • 0 kudos

if else condition task doubt

Hi community,The if else condition task couldn't be used as real if condition? Seems that if the condition goes to False the entire job will be stop. Is it a right behaviour?

  • 2753 Views
  • 1 replies
  • 0 kudos
Latest Reply
jeremy98
Honored Contributor
  • 0 kudos

Hi, I found that the problem is here: - task_key: get_email_infos max_retries: 3 min_retry_interval_millis: 150000 depends_on: - task_key: check_type_of_trigger outcome: "true" ...

  • 0 kudos
turagittech
by Contributor
  • 1305 Views
  • 3 replies
  • 1 kudos

External Table refresh

Hi,I have a blob storage area in Azure where json files are being created. I can create an external table on the storage blob container, but when new files are added I don't see extra rows to query the data. Is there a better approach to accessing th...

  • 1305 Views
  • 3 replies
  • 1 kudos
Latest Reply
Nivethan_Venkat
Contributor III
  • 1 kudos

Hi @turagittech,The above error indicates that your table seems to be in DELTA format. Please check the table creation statement, if the table format is JSON or DELTA.PS: By default, if you are not specifying any format while creating the table on to...

  • 1 kudos
2 More Replies
Walter_N
by New Contributor II
  • 864 Views
  • 2 replies
  • 0 kudos

Resolved! DLT pipeline task with full refresh once in a while

Hi all, I'm using Databricks workflow with some dlt pipeline tasks. These tasks requires a full refresh at some times due to schema changes in the source. I've been doing the full refresh manually or set the full refresh option in the job settings, t...

  • 864 Views
  • 2 replies
  • 0 kudos
Latest Reply
MariuszK
Valued Contributor III
  • 0 kudos

Hi,Did you check a possibility to use if/else task? You could define some scriteria and pass it from a notebok that will check if it's time for full refresh or just resfres.

  • 0 kudos
1 More Replies
scorpusfx1
by New Contributor II
  • 1138 Views
  • 4 replies
  • 0 kudos

Delta Live Table SCD2 performance issue

Hi Community,I am working on ingestion pipelines that take data from Parquet files (200 MB per day) and integrate them into my Lakehouse. This data is used to create an SCD Type 2 using apply_changes, with the row ID as the key and the file date as t...

Data Engineering
apply_change
dlt
SCD2
  • 1138 Views
  • 4 replies
  • 0 kudos
Latest Reply
Stefan-Koch
Valued Contributor II
  • 0 kudos

hi @scorpusfx1 What kind of source data do you have? Are these parquet files daily full snapshots of source tables? If so, you should use apply_changes_from_snapshot, which is exactly built for this use case. https://docs.databricks.com/aws/en/dlt/py...

  • 0 kudos
3 More Replies
analytics_eng
by New Contributor III
  • 4489 Views
  • 4 replies
  • 1 kudos

Connection reset by peer logging when importing custom package

Hi! I'm trying to import a custom package I published to Azure Artifacts, but I keep seeing the INFO logging below, which I don't want to display. The package was installed correctly on the cluster, and it imports successfully, but the log still appe...

  • 4489 Views
  • 4 replies
  • 1 kudos
Latest Reply
siklosib
New Contributor II
  • 1 kudos

What solved this problem for me is to remove the root logger configuration from the logging config and create another one within the loggers section. See below.{ 'version': 1, 'disable_existing_loggers': False, 'formatters': { 'simple...

  • 1 kudos
3 More Replies
nhuthao
by New Contributor II
  • 997 Views
  • 5 replies
  • 1 kudos

SQL is not enabled

Hi All,I have registered on Databricks successfully. However, SQL is not enabled.Please help me how to activate SQL.Thank you very much,

nhuthao_0-1741243500074.png
  • 997 Views
  • 5 replies
  • 1 kudos
Latest Reply
Stefan-Koch
Valued Contributor II
  • 1 kudos

@nhuthao How did you solved it? What was the problem?

  • 1 kudos
4 More Replies
cgrant
by Databricks Employee
  • 18097 Views
  • 3 replies
  • 4 kudos

What is the difference between OPTIMIZE and Auto Optimize?

I see that Delta Lake has an OPTIMIZE command and also table properties for Auto Optimize. What are the differences between these and when should I use one over the other?

  • 18097 Views
  • 3 replies
  • 4 kudos
Latest Reply
basit
New Contributor II
  • 4 kudos

Is this still valid answer in 2025 ? https://docs.databricks.com/aws/en/delta/tune-file-size#auto-compaction-for-delta-lake-on-databricks 

  • 4 kudos
2 More Replies

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local community—sign up today to get started!

Sign Up Now
Labels