- 1071 Views
- 3 replies
- 2 kudos
Resolved! how to avoid extra column after retry upon UnknownFieldException
With autoloader.option("cloudFiles.schemaEvolutionMode", "addNewColumns") I have done retry after getting org.apache.spark.sql.catalyst.util.UnknownFieldException: [UNKNOWN_FIELD_EXCEPTION.NEW_FIELDS_IN_FILE] Encountered unknown fields during par...
- 1071 Views
- 3 replies
- 2 kudos
- 2 kudos
Hi @Hubert-Dudek the input is csv. readStream reads csv with .option("cloudFiles.inferColumnTypes", "true"). then, df.toDF( ) is called to rename the column name. The original csv header has space, that's why error message has "test 1_2 Prime". The r...
- 2 kudos
- 4723 Views
- 5 replies
- 2 kudos
How to run a python task that uses click for CLI operations
Hey,in my application I am using click to facilitate CLI operations. It works locally, in notebooks, when scripts are run locally, but it fails in Databricks. I defined a task that, as an entrypoint, accepts the file where the click-decorated functio...
- 4723 Views
- 5 replies
- 2 kudos
- 2 kudos
Hey guys,I think I managed to find a workaround. I will leave it here for everyone that is seeking the same answers, including future me.What I did is basically this piece of code:def main(): try: assign_variants(standalone_mode=False) ...
- 2 kudos
- 1598 Views
- 4 replies
- 6 kudos
Resolved! ONLY PNG format is available for databricks dashboard table download
I couldn't download the data straight from databricks dashboards in csv format starting from last night (night of Dec 1st, 2025). The only format that is available right now is PNG. I've tried downloading the data on multiple browsers but only the PN...
- 1598 Views
- 4 replies
- 6 kudos
- 6 kudos
Hey @Advika you saya quick workaround is to right-click and download the CSV from there.What do you mean? Where? I am right clicking all over my dashboard widget and don't see CSV download option. Can you be more specific?
- 6 kudos
- 1920 Views
- 7 replies
- 3 kudos
Resolved! Declarative Pipelines: set Merge Schema to False
Dear Team!I want to prevent at a certain table that the schema is automatically updated. With plain strucutred streaming I can do the following:silver_df.writeStream \ .format("delta") \ .option("mergeSchema", "false") \ .option("checkpoi...
- 1920 Views
- 7 replies
- 3 kudos
- 3 kudos
It is automatic in DLT. If there are significant schema changes, you need to full refresh. Maybe consider storing everything (the whole JSON) in a single VARIANT column and unpacking only what is necessary later - this way you will have it under cont...
- 3 kudos
- 1431 Views
- 3 replies
- 2 kudos
Resolved! API Call to return more than 100 jobs
Hello I have around 150 jobs and this is likely to increase.I use this call to get all the jobs and write them into a list called json.My logic here is to match a name to a job id and run the job using the job id. response = requests.get(hostHTTPS, j...
- 1431 Views
- 3 replies
- 2 kudos
- 2004 Views
- 3 replies
- 5 kudos
Resolved! user standard serverless with asset bundle on Azure
Anyone running into issues with using standard serverless with Asset bundle we tried all options with below line performance_target: STANDARDbut it ignore above value and uses performance optimized cluster which is not expected any lead with ri...
- 2004 Views
- 3 replies
- 5 kudos
- 5 kudos
resources: jobs: my_dabs: performance_target: STANDARD Please check whether it is on the correct level in the YAML. Also consider updating the CLI. I've just tested it, and it worked properly.
- 5 kudos
- 997 Views
- 3 replies
- 4 kudos
Resolved! Getting below error when trying to create a Data Quality Monitor for the table. ‘Cannot create Monit
Getting below error when trying to create a Data Quality Monitor for the table.‘Cannot create Monitor because it exceeds the number of limit 500'.
- 997 Views
- 3 replies
- 4 kudos
- 4 kudos
Maybe this is not a situation, but trial accounts have lower quotas. Tried to find the quota related to the monitor in "databricks resource-quotas list-quotas" but couldn't find it.Your account contact in databricks can probably adjust it or find wit...
- 4 kudos
- 3796 Views
- 4 replies
- 4 kudos
Resolved! How to create a Unity Catalog Connection to SQL Server using service principal??
I am trying to use the Databricks Python SDK (v 0.63.0) to create a Unity Catalog connection to a Azure-hosted SQL Server database using an Azure service principal to authenticate. I have successfully done this via the Workspace UI, but I am trying t...
- 3796 Views
- 4 replies
- 4 kudos
- 4 kudos
Hi @ScottH ,You need to configure it in following way (I've tested it and it works). In a place where a red arrow is pointing you need to provide your own tenant_id:
- 4 kudos
- 4961 Views
- 5 replies
- 0 kudos
Unity Catalog is missing column in Catalog Explorer
I have just altered one of my tables and added a column.ALTER TABLE tpch.customer ADD COLUMN C_CUSTDETAILS struct<key:string,another_key:string,boolean_key:boolean,extra_key:string,int_key:long,nested_object:struct<more:long,arrayOne:array<string>>>A...
- 4961 Views
- 5 replies
- 0 kudos
- 0 kudos
I had this error just recently. I did basically the same table transformation 4 times over the course of 2 days. Added two new fields to two different tables using the same SQL syntax. It worked 3 out of 4 times, but for 1 the column is not available...
- 0 kudos
- 2245 Views
- 3 replies
- 2 kudos
Lakebridge code conversion from oracle to databricks sql
Hi Community,I’m working on migrating several Oracle views to SparkSQL using the Databricks Labs Lakehouse Bridge tool. I’m facing issues while converting the code .For oracle views and materialized views .Problems I’m encountering:The converted SQL...
- 2245 Views
- 3 replies
- 2 kudos
- 2 kudos
@Louis_Frolio - will there be similar guidelines for other 'source' code too such as T-SQL, Teradata. Any such comprhensive docs on Laeebridge to cover per source.
- 2 kudos
- 2394 Views
- 1 replies
- 2 kudos
Resolved! Moving files using DBUtils is so slow
I am using the platform DBUtils.fs.mv() on databricks clusters, and facing issues with move operation slowness.I move files in UC Volumes or ADLS storage abfss links, which work but is so slow.I mean it takes hours to transfer files that used to take...
- 2394 Views
- 1 replies
- 2 kudos
- 2 kudos
Hello @murtadha_s , here are some helpfult tips and hints to help you further diagnose the slowness. Totally expected behavior here: object-storage moves with dbutils.fs.mv will be much slower than HDFS. Under the hood, dbutils isn’t doing an atom...
- 2 kudos
- 871 Views
- 1 replies
- 0 kudos
Serverless Compute – ADLS Gen2 Authorization Failure with RBAC
We are facing an authorization issue when using serverless compute with ADLS Gen2 storage. Queries fail with:Code AbfsRestOperationException: Operation failed: "This request is not authorized to perform this operation.", 403 AuthorizationFailureDetai...
- 871 Views
- 1 replies
- 0 kudos
- 0 kudos
private link from serverless, as probably you are not allowing public internet access. Configure private connectivity to Azure resources - Azure Databricks | Microsoft Learn you need to add both dfs and blob
- 0 kudos
- 15431 Views
- 7 replies
- 3 kudos
Resolved! SQL Warehouse Serverless - Not able to access the external tables in the hive_metastore
I have DLT tables created under the hive_metastore with external data stored in ADL gen2.The ADL blob storage is mounted into /mnt/<storage-account>The tables are successfully created and accessible from my notebooks, as well the ADL storage.I have c...
- 15431 Views
- 7 replies
- 3 kudos
- 3 kudos
we can use terraform to create NCC (Network Connectivity Configuration). It will create a private endpoint on storage account and approve it manually because it is not auto approved.
- 3 kudos
- 2572 Views
- 2 replies
- 1 kudos
Resolved! Multiple Databricks Issues: Spark Context Limit, Concurrency Load, API Character Limit & Job Timeout
I am encountering multiple issues in our Databricks environment and would appreciate guidance or best-practice recommendations for each. Details below:1. [MaxSparkContextsExceeded] Too many execution contexts are open right now (Limit 150)Error: [Max...
- 2572 Views
- 2 replies
- 1 kudos
- 1 kudos
I would like to add my experience with 3. Databricks API 10k Character LimitWe had a similar issue, and this limit cannot be changed. Instead review concepts of sharing the input/output between Databricks and caller using cloud storage like ADLS. Pro...
- 1 kudos
- 2138 Views
- 4 replies
- 1 kudos
Reset committed offset of spark streaming to capture missed data
I have a very straightforward setup between Azure Eventhub and DLT using the kafka endpoint through spark streaming.There were network issues and the stream didn't pick up some event, but still progressed (and committed) the offset for some reasonAs ...
- 2138 Views
- 4 replies
- 1 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 582 | |
| 349 | |
| 287 |
