Databricks Community

The deprecation of the Enable libraries and init scripts on shared Unity Catalog clusters setting is in Databricks Runtime 16.0 and above. Please refer to the documentation here  for deprecation. Disabling this feature at the workspace level would pr...

I am wondering if it is using Remote result cache, in that case the config should work. There are 4 types of cache mentioned here https://docs.databricks.com/en/sql/user/queries/query-caching.html#types-of-query-caches-in-databricks-sql Local cache: ...

To reference job parameters in the inputs field, use the syntax {{job.parameters.<name>}}. Kindly refer to https://docs.databricks.com/en/jobs/for-each.html

Hi @gvvishnu , Thanks for your question. My understanding is that the Apache Spark `hash()` function implements the `org.apache.spark.sql.catalyst.expressions.Murmur3Hash` expression. You can see this in the Spark source code here: https://github.com...

No, Private link is to set up your workspace with no access to the internet, have you tried allowing the NCC ips with on the on Prem firewall?

Hi @Greg_c ,In Databricks Asset Bundles you have a possibility to pass parameter to SQL File Task.Here is end to end example:1. My SQL File (with :id parameter): 2. The job YAML:resources: jobs: run_sql_file_job: name: run_sql_file_job ...

I have found that only progressbar2 can work in both interactive notebooks and workflow notebooks. It's limited, but better than nothing. Tqdm is broken in workflows.

Hey @Kayla ,This is because serverless infrastructure is fully managed by Databricks, and you do not have direct control over the underlying resources as you do with standard clusters (non-serverless SQL warehouse)You can track SQL Warehouse usage wi...

In case someone happens upon this in the future, I ended up using Unity Catalog with Hive metastore federation for Glue. The Iceberg support is currently "coming soon in Public Preview."

Thanks for the reply.Its working in dbr 15.4 but I want to use it with 13.3, is there a workaround?

Hey Greg_cI use DBT daily for batch data ingestion, and I believe it’s a great option. However, it’s important to consider that adopting DBT introduces additional complexity, and the team should carefully evaluate the impact of adding a new tool to t...

Hey @hboleto It’s difficult to accurately estimate the final cost of a Serverless cluster, as it is fully managed by Databricks. In contrast, Classic clusters allow for finer resource tuning since you can define spot instances and other instance type...

You can do all of this yourself with a testing workflow.  You can create your data in a notebook or keeping a backup copy of tables, and copy them fresh for your tests.  This would be the first step of the workflow.  Then call your notebooks.  Your c...