Databricks Community

Ping @Alberto_Umana 

Hi @dbx-user7354 ,OrderBy() should perform a global sort as showed in plot-2, but as per your problem it is sorting the data within the partitions which is the behavior of sortWithinPartitions(), so to solve this error. Please try with the latest DBR...

Permission denied appears to be the cause of the error message. To read from the DBFS path and write to the workspace folder, please make sure you have the required permissions. The following permissions may be required:The DBFS file path can be read...

The deprecation of the Enable libraries and init scripts on shared Unity Catalog clusters setting is in Databricks Runtime 16.0 and above. Please refer to the documentation here  for deprecation. Disabling this feature at the workspace level would pr...

This can be an issue with authentication or configuration being missing. When constructing the AccountClient class instance, please ensure that the required authentication details are present. Additionally, since this action is account-level, make su...

I am wondering if it is using Remote result cache, in that case the config should work. There are 4 types of cache mentioned here https://docs.databricks.com/en/sql/user/queries/query-caching.html#types-of-query-caches-in-databricks-sql Local cache: ...

To reference job parameters in the inputs field, use the syntax {{job.parameters.<name>}}. Kindly refer to https://docs.databricks.com/en/jobs/for-each.html

Hi @gvvishnu , Thanks for your question. My understanding is that the Apache Spark `hash()` function implements the `org.apache.spark.sql.catalyst.expressions.Murmur3Hash` expression. You can see this in the Spark source code here: https://github.com...

No, Private link is to set up your workspace with no access to the internet, have you tried allowing the NCC ips with on the on Prem firewall?

Hi @Greg_c ,In Databricks Asset Bundles you have a possibility to pass parameter to SQL File Task.Here is end to end example:1. My SQL File (with :id parameter): 2. The job YAML:resources: jobs: run_sql_file_job: name: run_sql_file_job ...

Guess you will find some differences in before and after, such as the use of the wildcard to point to folders in ADLS2 for external tables is supported in hive but not in UC catalogs.

Hello,There are two ways to determine whether a DLT pipeline is running in Full Refresh or Incremental mode:DLT Event Log SchemaThe details column in the DLT event log schema includes information on "full_refresh". You can use this to identify whethe...