I am noticing a difference between using autoloader in an interactive notebook vs using it in a Spark Declarative Pipeline (DLT Pipeline). This issue seems to be very similar to this other unanswered post from a few years ago. Bug report: the delimit...
Hello,When running on a serverless cluster in DLT our notebook first tries to install some python whls onto the cluster. We have noticed that when in development and running a pipeline many times over in a short space of time between runs that the pi...
Hello,I am trying to set up autoloader using file notifications but as the storage account we are reading from is a premium storage account we have setup event subscriptions to pump the blob events to queues that exist on a standard gen 2 storage acc...
Hello, I am trying to clarify if Databricks Claude 3.7 Native serving endpoint is safe for use in our enterprise or if it needs to be disabled.I am trying to understand if the data sent to this endpoint is kept within our geo region (UKSouth/Europe)....
Hello, With the recent changes to Delta Live Tables, I was wondering when the python stub will be updated to reflect the new methods that are available ?Link to the Pypi repo:databricks-dlt·PyPI
Hey, I agree it would be ideal to have the data on a storage account that supports queues but unfortunately this is not in my control.Regarding your option:Use Event Hubs or Service Bus instead of Storage Queues: - Reconfigure Event Grid: Change your...
Hey Lou,Thanks for clarifying and for the swift response. Though nothing you said was news to me it was beneficial to have you confirm that what I am asking for is currently not possible and your alternative is currently what we are doing  Thanks,Ch...
Hey @Panda ,That will work but when you want to do this for each of the columns in your table it becomes very unclean in comparison to using something like the StructField Metadata attribute.At the moment you would end up doing somethings like: de...
