Databricks Community

Option C will be the right answer in my perspective. Though a job ownership is transferred to another user, creater_user field will never change. In fact, "Creater User" field in job details panel is non-editable field.If I am wrong, clarification wi...

Hey @DBUser2 It looks that delete is not supported at all based on documentation: Write-backThe Simba Apache Spark ODBC Connector supports translation for the followingsyntax when connecting to a Spark Thrift Server instance that is running Spark 1.3...

Integrating a Databricks notebook with a Git repository using a feature branch sounds like a clean workflow for version control. The step-by-step process makes it straightforward to collaborate and track changes effectively. It's great that Databrick...

Hi @tgburrin-afs, @Mounika_Tarigop ,As I understand the question is about running concurrent tasks within a single job rather than running concurrent jobs. max_concurrent_runs controls how many times a whole job can run simultaneously, not the concur...

To add a CreatedDate column that captures the timestamp when a record is first inserted into the table, you can modify your Delta Live Tables (DLT) pipeline setup as follows: 1) Define the schema for your streaming table to include the CreatedDate co...

In DBFS, the checkpoint directory is automatically created when you set it using spark.sparkContext.setCheckpointDir(checkpoint_dir). This means that you do not need to explicitly create the directory beforehand using dbutils.fs.mkdirs(checkpoint_dir...

There are no specific column datatype constraints for the first 32 columns used for the statistics that help with data skipping in Databricks. However, data skipping is not supported for partition columns.

Delta Sharing in Databricks does support caching, specifically through the use of Delta Cache.Delta cache will be enabled by default on the recipient cluster. Delta Cache helps improve the performance of data reads by storing a copy of the data on th...

Now the Python UDF support is currently in public preview, and it is supported in the current channel as well. Could you please try to re-run the code.  https://docs.databricks.com/en/delta-live-tables/unity-catalog.html

You can separate the user interface (UI) from the code logic by using two notebooks. 1) The first notebook will contain the code that performs the actions based on the widget inputs. 2) The second notebook will contain the widgets that users interact...

To achieve "PARTITION OVERWRITE" instead of "COMPLETE RECOMPUTE" when running a Delta Live Table (DLT) materialized view, you need to configure your pipeline to use incremental refresh. This can be done by setting up your DLT pipeline to use serverle...

Delta Live Tables (DLT) can indeed process multiple streaming tables within a single pipeline.Here are a few things to check:1) Verify that each streaming table has a unique checkpoint location. Checkpointing is crucial for maintaining the state of s...