Databricks Community

Integrating a Databricks notebook with a Git repository using a feature branch sounds like a clean workflow for version control. The step-by-step process makes it straightforward to collaborate and track changes effectively. It's great that Databrick...

In DBFS, the checkpoint directory is automatically created when you set it using spark.sparkContext.setCheckpointDir(checkpoint_dir). This means that you do not need to explicitly create the directory beforehand using dbutils.fs.mkdirs(checkpoint_dir...

There are no specific column datatype constraints for the first 32 columns used for the statistics that help with data skipping in Databricks. However, data skipping is not supported for partition columns.

Delta Sharing in Databricks does support caching, specifically through the use of Delta Cache.Delta cache will be enabled by default on the recipient cluster. Delta Cache helps improve the performance of data reads by storing a copy of the data on th...

Now the Python UDF support is currently in public preview, and it is supported in the current channel as well. Could you please try to re-run the code.  https://docs.databricks.com/en/delta-live-tables/unity-catalog.html

You can separate the user interface (UI) from the code logic by using two notebooks. 1) The first notebook will contain the code that performs the actions based on the widget inputs. 2) The second notebook will contain the widgets that users interact...

To achieve "PARTITION OVERWRITE" instead of "COMPLETE RECOMPUTE" when running a Delta Live Table (DLT) materialized view, you need to configure your pipeline to use incremental refresh. This can be done by setting up your DLT pipeline to use serverle...

Delta Live Tables (DLT) can indeed process multiple streaming tables within a single pipeline.Here are a few things to check:1) Verify that each streaming table has a unique checkpoint location. Checkpointing is crucial for maintaining the state of s...

Auto loader's scope is limited to incrementally loading files from storage, and there is no such functionality to just load the latest file from a group of files, you'd likely want to have this kind of "last updated" logic in a different layer or in ...

@szymon_dybczak is it possible to do directly in Silver layer? We do not have option to go fo Gold layer. I tried to create a TEMP VIEW in Silver DLT pipeline but it gives error for circular dependency as I am comparing data from Silver it self and a...

To answer the original question, autoloader does not use locks when reading files. You are however limited by the underlying storage system, ADLS in this example. Going by what has been mentioned (long batch times, but spark jobs finish really fast) ...

If the files have already been loaded by autoloader (like same name and path), this can be tricky. I recommend starting a separate autoloader stream and specifying filters on it to match your start and end dates. If you'd instead like to rely on the ...

Thanks for your question! Although it shouldn't be necessary, could you please try the following: Set the spark.databricks.sql.initial.catalog.name configuration to my_catalog in your Spark session to ensure the correct catalog is initialized. Use ...