Databricks Community

DazzaiDe

We’re currently designing our Medallion Architecture pipelines using Lakeflow Jobs, and I wanted to get some opinions on orchestration best practices.Right now, our approach is essentially 1 job per target table (for example, each Bronze/Silver/Gold ...

DazzaiDe

Is there any way a job can be triggered with overwritten files through a file trigger?I knew the limitations, but I just want to have other opinions or solutions.LimitationsOnly new files trigger runs. Overwriting an existing file with a file of the ...

DazzaiDe

I’m looking to follow best practices with Databricks Auto Loader. When ingesting different file formats, is it considered good practice to always read data as strings first, or is it better to use format-specific readers (e.g., JSON, CSV, binary) fro...

DazzaiDe

Hi @Ashwin_DSA ,Btw, my trigger is pointed to a root path already. My goal is to reprocess any file (if they reupload the raw file again because of some fixes to the data). It's possible with autoloader (reprocess overwritten files), but my problem i...

Databricks Community

User Stats

User Activity

Best Practices: 1 job per 1 target table

Lakeflow jobs file trigger thru overwritten files

Best practices for using autoloader

Re: Lakeflow jobs file trigger thru overwritten files