Databricks Community

I am trying to run a DLT job that uses GraphFrames, which is in the ML standard image.   I am using it successfully in my job compute instances but I'm running into problems trying to use it in a DLT job.  Here are my overrides for the standard job c...

I am trying to run a DLT job that uses GraphFrames, which is in the ML standard image.   I am using it successfully in my job compute instances.  Here are my overrides for the standard job compute policy: {"spark_version": {"type": "unlimited","defau...

I'm experimenting with several approaches to implement an incremental autoloader query either in DLT or in a pipeline job.   The complexities:- Moving approximately 30B records from a nasty set of nested folders on S3 in several thousand csv files.  ...

The databricks guide outlines several use cases for loading data via delta live tables:https://docs.databricks.com/en/delta-live-tables/load.html This includes autoloader from cloudfiles, kafka messages, small static datasets, etc.But, one use case i...