Databricks Community

Hi everyone, in our project we are trying to implement liquid clustering. We are testing liquid clustering with a test table called status_update, where we need to update the status for different market IDs. We are trying to update the status_update ...

Hello @ble! This was a known issue that the engineering team had been investigating. Could you kindly confirm whether you’re still experiencing the issue or if it has been resolved?

cloudFiles.cleanSource.retentionDurationType: Interval StringAmount of time to wait before processed files become candidates for archival with cleanSource. Must be greater than 7 days for DELETE. No minimum restriction for MOVE.Available in Databrick...

Hi @Akshay_Petkar Migrating Amazon Redshift SQL to Databricks (especially Delta Lake or Unity Catalog-backed systems) using BladeBridge is a practical yet less-documented use case.Since BladeBridge is a commercial tool with limited public documentati...

Thank you very much for your response and explanation.Unfortunately, this takes me back to the original error. I have tried changing the set_tracking_uri to my workspace url and I get the same error.Any ideas what this could be?

Hey @Datagyan ,If your query result is larger than 25MB, Databricks automatically uses disposition=EXTERNAL_LINKS, which returns the result in multiple chunked files(external links).Currently, there's no option to get a single file directly from the ...

You're encountering a common issue when using service principals and job clusters with workspace-scoped paths. This typically happens due topermission mismatches or cluster identity issues. Here’s a breakdown of the root cause and a recommended solut...

For your consideration: To address the challenge of passing message handles from executors back to the driver within the DataSourceStreamReader, consider the following approaches: Challenges in Spark Architecture 1. Executor Memory Isolation: Execut...

Here are some things considerations:   The errors experienced in your production streaming jobs—ERR_NGROK_3200 and Spark driver failed to start within 900 seconds—stem from distinct causes related to connectivity, underlying system constraints, and d...

@Sidhant07 also using Kylin4 and Sqoop I am getting: "Unsupported transaction isolation level: 2" when running sqoop list-tables. 

I just accomplished by modifying the DatabricksClient by adding the feature_store_uri parameter to it and passing it to each call of get_host_creds. Lets see if databricks releases this simple change soon...