Databricks Community

ChrisLawford_n1 · 01-28-2026

Hello, I am using databricks autoloader with managedfileevents turned on and include existing files.I want to understand if there is a way of increasing the speed of the initial listing of the files for autoloader.I thought that the idea behind the m...

ChrisLawford_n1 · 01-15-2026

I am noticing a difference between using autoloader in an interactive notebook vs using it in a Spark Declarative Pipeline (DLT Pipeline). This issue seems to be very similar to this other unanswered post from a few years ago. Bug report: the delimit...

ChrisLawford_n1 · 10-27-2025

Hello,When running on a serverless cluster in DLT our notebook first tries to install some python whls onto the cluster. We have noticed that when in development and running a pipeline many times over in a short space of time between runs that the pi...

ChrisLawford_n1 · 07-29-2025

Hello,I am trying to set up autoloader using file notifications but as the storage account we are reading from is a premium storage account we have setup event subscriptions to pump the blob events to queues that exist on a standard gen 2 storage acc...

ChrisLawford_n1 · 07-23-2025

Hello, I am trying to clarify if Databricks Claude 3.7 Native serving endpoint is safe for use in our enterprise or if it needs to be disabled.I am trying to understand if the data sent to this endpoint is kept within our geo region (UKSouth/Europe)....

ChrisLawford_n1 · 02-05-2026

Hey Louis,Yeah Your explanation of each autoloader instance being responsible for its own position is what hit home. We are just unfortunate that the autoloader instances we have contain glob patterns that overlap a lot so the indexing to the outside...

ChrisLawford_n1 · 01-19-2026

Hey,Okay thanks @nikhilj0421. I have now solved the issue but not with a full refresh of the table. I had tried this previously and even deleted the DLT pipeline hoping that would provide me the clean slate if this lingering schema was an issue but w...

ChrisLawford_n1 · 08-20-2025

Do you have a mechanism to do this ?Any aspects that are accessible to allow this to be possible?

ChrisLawford_n1 · 08-20-2025

Any progress on this ? Did you find a solution ?

ChrisLawford_n1 · 07-29-2025

Hey, I agree it would be ideal to have the data on a storage account that supports queues but unfortunately this is not in my control.Regarding your option:Use Event Hubs or Service Bus instead of Storage Queues: - Reconfigure Event Grid: Change your...

Databricks Community

User Stats

User Activity

DeltaFileOperations: Listing improvement?

Bug Report: SDP (DLT) with autoloader not passing through pipe delimiter/separator

Network error on subsequent runs using serverless compute in DLT

Autoloader with file notifications on a queue that is on a different storage account to the blobs

Clarification on Databricks Claude 3.7 Sonnet native serving endpoint data processing location.

Re: DeltaFileOperations: Listing improvement?

Re: Bug Report: SDP (DLT) with autoloader not passing through pipe delimiter/separator

Re: Move delta live tables amongst schemas within a catalog

Re: Move managed DLT table from one schema to another schema in Databricks

Re: Autoloader with file notifications on a queue that is on a different storage account to the blob