Databricks Community

Spark configs are limited in Serverless, this are the supported configs you can set https://docs.databricks.com/en/release-notes/serverless/index.html#supported-spark-configuration-parameters 

For these scenarios, you can use schema evolution capabilities like mergeSchema or opt to use the new VariantType to avoid requiring a schema at time of ingest.

For this style of ETL, there are 2 methods. The first method, strictly for partitioned tables, is Dynamic Partition Overwrites, which require a Spark configuration to be set and detect which partitions that are to be overwritten by scanning the input...

it appears that Databricks no longer offers downloadable notebooks or DBC files for self-paced courses, including the "Data Engineer Learning Plan."  However, if you are interested in working on labs within a provided Databricks environment, you can ...

Sure, I will inform the team in charge of it to review it.

Also, you need to pass the parameters using the --vars flag like that: dbt run --vars '{"start_time": "{{job.start_time.[timestamp_ms]}}"}' You will need to modify the 3rd dbt command in your job.

Hello @Akshith_Rajesh, In order to connect Trino to Unity Catalog HMS, please use this configuration instead: hive.metastore.uri=https://<DATABRICKS_HOST>:443/api/2.0/unity-hms-proxy/metadata hive.metastore.http.client.bearer-token=${ENV:DATABRICKS_T...

Hi Dfrozo,Great article. I liked how you wrote and explained everything in a very simplistic manner. I wish to read more from you.One comment. One of the Link in the blog is broken.In the sentence: "Privacera: Integrates with Databricks Unity Catalog...

Can you try creating another profile instead of the Default one and try with it, it seems that what it is not collecting is the cluster details but wanted to check with a new profile  

What would be the result if instead of using ' ' you use ` `? 

Got it, allow me with some time to look for information around this one

Hi @jeremy98, I think it has to do with the serverless version being used outside the workflow versus in DABs, since python version changes. please see: https://docs.databricks.com/en/release-notes/serverless/index.html both the versions have differe...

Is a special user account created by Databricks for managing system-level operations and configurations. This user is not listed in the admin dashboard of all workspace users because it is not a regular user account but rather a system account used i...