Databricks Community

To reference job parameters in the inputs field, use the syntax {{job.parameters.<name>}}. Kindly refer to https://docs.databricks.com/en/jobs/for-each.html

Hi @gvvishnu , Thanks for your question. My understanding is that the Apache Spark `hash()` function implements the `org.apache.spark.sql.catalyst.expressions.Murmur3Hash` expression. You can see this in the Spark source code here: https://github.com...

No, Private link is to set up your workspace with no access to the internet, have you tried allowing the NCC ips with on the on Prem firewall?

Hi @Greg_c ,In Databricks Asset Bundles you have a possibility to pass parameter to SQL File Task.Here is end to end example:1. My SQL File (with :id parameter): 2. The job YAML:resources: jobs: run_sql_file_job: name: run_sql_file_job ...

Just some clarification:  the caller notebook can be found with no issues, no matter the task's source is GIT or WORKSPACE.  However, the callee notebook, which is called by the caller notebook with dbutils.notebook.run(), cannot be found if the call...

I have found that only progressbar2 can work in both interactive notebooks and workflow notebooks. It's limited, but better than nothing. Tqdm is broken in workflows.

Hey @Kayla ,This is because serverless infrastructure is fully managed by Databricks, and you do not have direct control over the underlying resources as you do with standard clusters (non-serverless SQL warehouse)You can track SQL Warehouse usage wi...

In case someone happens upon this in the future, I ended up using Unity Catalog with Hive metastore federation for Glue. The Iceberg support is currently "coming soon in Public Preview."

Thanks for the reply.Its working in dbr 15.4 but I want to use it with 13.3, is there a workaround?

Hey Greg_cI use DBT daily for batch data ingestion, and I believe it’s a great option. However, it’s important to consider that adopting DBT introduces additional complexity, and the team should carefully evaluate the impact of adding a new tool to t...

Hey @hboleto It’s difficult to accurately estimate the final cost of a Serverless cluster, as it is fully managed by Databricks. In contrast, Classic clusters allow for finer resource tuning since you can define spot instances and other instance type...

You can do all of this yourself with a testing workflow.  You can create your data in a notebook or keeping a backup copy of tables, and copy them fresh for your tests.  This would be the first step of the workflow.  Then call your notebooks.  Your c...

Hi @Divya_sreeE you can pass dynamic variables between tasks using Databricks' job parameters.1) In your first Python wheel task, generate the dynamic variables and use the Databricks REST API to update the job parameters.2) In the For Each loop, ret...