Databricks Community

charl-p-botha

Dear community,According to [1] and other sources, all workspace users are assigned `CAN_CREATE` on lakebase projects, and this permission "can't be revoked".The problem is that such a project comes with by default a 8 - 16 CU lakebase compute instan...

charl-p-botha · 11-15-2024

Great blog post: https://community.databricks.com/t5/technical-blog/integrating-apache-spark-with-databricks-unity-catalog-assets/ba-p/97533I have attempted to reproduce this with Azure Databricks, and ADLS gen2 as the storage backend.Although I'm ab...

charl-p-botha · 11-06-2024

Dear DataBricks people,We are currently measuring DLT performance and cost on a medallion architecture with 150 to 300 tables, and we're interested in adding even more tables.I've been doing automated incremental streaming DLT pipelines every 3 hours...

charl-p-botha

Thank you very much for also looking into this.I've submitted this as feedback via the new Lakebase Postgres web-ui. It doesn't look like I can link to that submitted feedback here. Let's hope for the best.

charl-p-botha · 11-20-2024

Thanks @dkushari I looked at the github issue you posted, but it has to do specifically with DELTA_UNSUPPORTED_SCHEMA_DURING_READ when streaming *from* a delta table.The specific error I'm seeing is a key error for the Azure storage account hosting t...

charl-p-botha · 11-20-2024

Dear @dkushari From an external Spark system, I can now read from a kafka topic and write into a UC-managed external table (underlying storage is Azure) that I pre-created in a Databricks notebook.However, when I try to do exactly the same with readS...

charl-p-botha · 11-18-2024

@dkushari I hope that you can help me again.I'm now trying the external table path, but Databricks UC refuses to give me the necessary temporary credentials for the storage location. Please see https://github.com/unitycatalog/unitycatalog/issues/560#...

charl-p-botha · 11-15-2024

Aaah, the link you shared has the following:"Only tables are supported during the public preview.External tables support read and write.Managed tables can only be read."... and I was trying to append to a managed table.That is a plausible explanation...

Databricks Community

User Stats

User Activity

How to prevent users from creating Lakebase compute?

Error "Integrating Apache Spark with Databricks Unity Catalog Assets via Open APIs" on Azure

Thank you for the "setting up tables" speed-up from dlt release 2024.42.rc0 to 2024.44.rc1

Re: How to prevent users from creating Lakebase compute?

Re: Error "Integrating Apache Spark with Databricks Unity Catalog Assets via Open APIs" on

Re: Error "Integrating Apache Spark with Databricks Unity Catalog Assets via Open APIs" on

Re: Error "Integrating Apache Spark with Databricks Unity Catalog Assets via Open APIs" on

Re: Error "Integrating Apache Spark with Databricks Unity Catalog Assets via Open APIs" on