Databricks Community

Hello everyone,I’m currently working on a setup where my unprocessed real-time data arrives as .json files in Azure Data Lake Storage (ADLS). Every x minutes, I use Databricks Autoloader to pick up the new data, run my ETL transformations, and store ...

Hello everyone!I’m currently working on a project with shared functionalities across different Databricks bundles. I have separate folders for each bundle, along with a common libs/ folder that holds some Python modules intended to be shared across b...

Hello,I'm currently working on a Databricks Assets Bundle (DAB) that builds and deploys a Python wheel package. My goal is to deploy this package to a Volume so that other DAB jobs can use this common library.I followed the documentation and successf...

I'm using AutoLoader to process any new file or update that arrives to my landing area. And then I schedule the job using DB workflows to trigger on file arrival. The issue is that the trigger only executes when new files arrive, not when an exiting ...