Databricks Community

Databricks introduced Managed File Events which completely bypasses the need for the cluster's identity to provision cloud resources, resolving the conflict with the Shared cluster mode.Steps to Implement in DLTEnable File Events on the External Loca...

Yes, for your setup, Databricks Auto Loader will create a separate event queue for each independent stream running with the cloudFiles.useManagedFileEvents = true option.As you are running - 1 stream per table, 1 unique directory per stream and 1 uni...

Found an answer!Basically pip was somehow installed the dependencies from the git repo first and was not following the given order so in order to solve this, I added the libraries for conda to install.``` conda_env = { "channels": [ "defa...

The Lakebridge analyzer counts DDL statements, not regular queries. A simple SELECT * is DML, not DDL, so it won’t show up in the DDL count.To get meaningful results for SnowSQL code analysis:Include actual DDL statements like CREATE TABLE, ALTER TAB...

Thanks for the detailed context—here’s a concise, actionable troubleshooting plan tailored to Databricks with Unity Catalog volumes and Avro + Confluent Schema Registry over APIM with mTLS. What’s likely going wrong Based on your description, the ini...

It looks like the issue is with the Azure credential chain rather than Service Bus itself; in Databricks notebooks, DefaultAzureCredential won’t succeed unless there’s a valid identity available (env vars, CLI login, managed identity, or a Databricks...

Thanks for the details. The PyRFC package is a Python binding around the SAP NetWeaver RFC SDK and requires the SAP NW RFC SDK to be present at build/run time; it does not work as a pure Python wheel on Linux without the SDK. The project is archived ...

You cannot add a Microsoft Entra ID–managed service principal to Databricks on AWS today; AWS workspaces only support Databricks‑managed service principals that you create in the Databricks account/workspace, not service principals federated from Ent...

Hi @Nidhig ,1. Regarding pipeline cost - here you're mostly paying for compute usage. So the exact price depends on which plan you are at and which cloud provider you are using. For instance, for Azure premium plan and US East region you have followi...

Thanks for weighing in. Are you saying CREATE OR REFRESH STREAMING TABLE user.demo.test_bronze_st  cannot be used in FREE Edition?If we can use it, how do to create STREAM read_files('/Volumes/xxx_ws/demo/raw_files/test.csv'),where .csv sitting on lo...

Here are few DQ packages for DLT or LDP that you can try.1. Databricks Labs DQXPurpose-built for Spark and Databricks.Rule-based checks on DataFrames (batch & streaming).Supports quarantine and profiling.Lightweight and easy to integrate.2. Great Exp...