Databricks Community

ChrisLawford_n1

Hello,I am trying to use delta live tables in a production setting but I am having an issue in ensuring that I will be able to confirm the status of the data that the various tables have processed in the pipeline.In the most basic case let me imagine...

ChrisLawford_n1

I would like to get a recommendation on how to structure ingestion of lots of tables of data. I am using autoloader currently with the directory searching mode.I have concerns about performance in the future and have a requirement to ensure that data...

ChrisLawford_n1

Hey erigaud,You can pass information into a DLT pipleine in a similar way to how you can pass information in to a notebook when being run in a job. In the DLT pipeline case you can set values in the configuration. Configure pipeline settings for Delt...

ChrisLawford_n1

Hey,Thanks for the swift reply, I don't have control over the way the source data is loaded into the blob store. So as I understand your answer it would be to go with the first option I listed:have an intermediary service that is triggered by the kaf...

Databricks Community

User Stats

User Activity

Delta Live Tables: How to determine what batch you have processed up to?

Autoloader configuration for multiple tables from the same directory

Re: DLT - Unity catalog and volume - Dynamically access volume path

Re: Autoloader configuration for multiple tables from the same directory