Databricks Community

Hi @Mirza1 , Greetings!  Can you please confirm if it is an ADLS gen2 table? If yes then can you please give it a try to run the table schema by setting spark configs for gen2 at the cluster level?   You can refer to this document to set the spark co...

Had to hold on this testing. But, when I get a chance to work on this, I will update my findings. Thank you!

@PerformanceTest - can you please check your DB workspace terraform script to see if there is a different CNAME defined for your host workspace.  

@ByteForge - Kindly raise a support case with Databricks to work with Engg to increase the limits for your workspace.

Hi @databricksdev,  Azure Databricks applies default tags to each cluster, including Vendor, Creator, ClusterName, and ClusterId. In addition, it applies two default tags on job clusters: RunName and JobId1. However, these tags are only applied to...

MapType does not maintain order (json itself too).Can you apply the ordering yourself afterwards?

This can be resolved by redefining the schema structure explicitly and using that schema to read the file. from pyspark.sql.types import StructType, StructField, StringType, IntegerType, ArrayType# Define the schema according to the JSON structuresch...

Hi @Ajay-Pandey ,thank you for the hints. I will try to recreate the job via UI. I ran the tasks in a Github workflow. The file locations are mixed: the first two tasks (python and dlt) are located in the databricks/src folder. The dbt files come fro...