Databricks Community

This can be an issue with authentication or configuration being missing. When constructing the AccountClient class instance, please ensure that the required authentication details are present. Additionally, since this action is account-level, make su...

I am wondering if it is using Remote result cache, in that case the config should work. There are 4 types of cache mentioned here https://docs.databricks.com/en/sql/user/queries/query-caching.html#types-of-query-caches-in-databricks-sql Local cache: ...

To reference job parameters in the inputs field, use the syntax {{job.parameters.<name>}}. Kindly refer to https://docs.databricks.com/en/jobs/for-each.html

Hi @gvvishnu , Thanks for your question. My understanding is that the Apache Spark `hash()` function implements the `org.apache.spark.sql.catalyst.expressions.Murmur3Hash` expression. You can see this in the Spark source code here: https://github.com...

No, Private link is to set up your workspace with no access to the internet, have you tried allowing the NCC ips with on the on Prem firewall?

Hi @Greg_c ,In Databricks Asset Bundles you have a possibility to pass parameter to SQL File Task.Here is end to end example:1. My SQL File (with :id parameter): 2. The job YAML:resources: jobs: run_sql_file_job: name: run_sql_file_job ...

Guess you will find some differences in before and after, such as the use of the wildcard to point to folders in ADLS2 for external tables is supported in hive but not in UC catalogs.

Hello,There are two ways to determine whether a DLT pipeline is running in Full Refresh or Incremental mode:DLT Event Log SchemaThe details column in the DLT event log schema includes information on "full_refresh". You can use this to identify whethe...

Just some clarification:  the caller notebook can be found with no issues, no matter the task's source is GIT or WORKSPACE.  However, the callee notebook, which is called by the caller notebook with dbutils.notebook.run(), cannot be found if the call...

I have found that only progressbar2 can work in both interactive notebooks and workflow notebooks. It's limited, but better than nothing. Tqdm is broken in workflows.

Hey @Kayla ,This is because serverless infrastructure is fully managed by Databricks, and you do not have direct control over the underlying resources as you do with standard clusters (non-serverless SQL warehouse)You can track SQL Warehouse usage wi...

In case someone happens upon this in the future, I ended up using Unity Catalog with Hive metastore federation for Glue. The Iceberg support is currently "coming soon in Public Preview."

Thanks for the reply.Its working in dbr 15.4 but I want to use it with 13.3, is there a workaround?

Hey Greg_cI use DBT daily for batch data ingestion, and I believe it’s a great option. However, it’s important to consider that adopting DBT introduces additional complexity, and the team should carefully evaluate the impact of adding a new tool to t...