Databricks Community

charl-p-botha · 11-15-2024

Great blog post: https://community.databricks.com/t5/technical-blog/integrating-apache-spark-with-databricks-unity-catalog-assets/ba-p/97533I have attempted to reproduce this with Azure Databricks, and ADLS gen2 as the storage backend.Although I'm ab...

charl-p-botha · 11-06-2024

Dear DataBricks people,We are currently measuring DLT performance and cost on a medallion architecture with 150 to 300 tables, and we're interested in adding even more tables.I've been doing automated incremental streaming DLT pipelines every 3 hours...

charl-p-botha · 11-20-2024

Thanks @dkushari I looked at the github issue you posted, but it has to do specifically with DELTA_UNSUPPORTED_SCHEMA_DURING_READ when streaming *from* a delta table.The specific error I'm seeing is a key error for the Azure storage account hosting t...

charl-p-botha · 11-20-2024

Dear @dkushari From an external Spark system, I can now read from a kafka topic and write into a UC-managed external table (underlying storage is Azure) that I pre-created in a Databricks notebook.However, when I try to do exactly the same with readS...

charl-p-botha · 11-18-2024

@dkushari I hope that you can help me again.I'm now trying the external table path, but Databricks UC refuses to give me the necessary temporary credentials for the storage location. Please see https://github.com/unitycatalog/unitycatalog/issues/560#...

charl-p-botha · 11-15-2024

Aaah, the link you shared has the following:"Only tables are supported during the public preview.External tables support read and write.Managed tables can only be read."... and I was trying to append to a managed table.That is a plausible explanation...

charl-p-botha · 11-15-2024

I was definitely logged in. The error message I got each time as I tried to leave a comment was:"Correct the highlighted errors and try again.The message subject contains <TM symbol>, which is not permitted in this community. Please remove this conte...

Databricks Community

User Stats

User Activity

Error "Integrating Apache Spark with Databricks Unity Catalog Assets via Open APIs" on Azure

Thank you for the "setting up tables" speed-up from dlt release 2024.42.rc0 to 2024.44.rc1

Re: Error "Integrating Apache Spark with Databricks Unity Catalog Assets via Open APIs" on

Re: Error "Integrating Apache Spark with Databricks Unity Catalog Assets via Open APIs" on

Re: Error "Integrating Apache Spark with Databricks Unity Catalog Assets via Open APIs" on

Re: Error "Integrating Apache Spark with Databricks Unity Catalog Assets via Open APIs" on

Re: Error "Integrating Apache Spark with Databricks Unity Catalog Assets via Open APIs" on