Databricks Community

Hi @naman0012, There are a few different approaches depending on where these "views" live and what your exact architecture looks like. Let me walk through the options. CLARIFYING THE SCENARIO The key question is: are these database views on an extern...

Hi @souravroy1990, You are correct that Unity Catalog tags (set via ALTER TABLE SET TAGS or ALTER TABLE ALTER COLUMN SET TAGS) are not propagated to recipients through Delta Sharing. Tags are treated as Unity Catalog governance metadata that lives in...

Hi @tdata, The approach you are using (PyIceberg with the Unity Catalog Iceberg REST catalog) does support writes to managed Iceberg tables, so the pattern is correct. The SSL error you are seeing is almost certainly caused by running this from withi...

Hi @ajay_wavicle, There are a couple of well-established patterns for managing Databricks resources with the Terraform provider and CLI without running everything locally. Here is a breakdown of the options and recommended approach. WHERE TO RUN TERR...

Hi @ajay_wavicle, There are several approaches you can use to extract read/source paths from notebooks, including Auto Loader (cloudFiles) paths. The right choice depends on whether you want runtime lineage data or static code analysis. APPROACH 1: U...

Hi @Raghav1, There are a few things happening here, so let me walk through each one and give you a path forward. UNDERSTANDING THE GATEWAY_DEFINITION ERROR The error "Modifying following parameter gateway_definition in pipeline settings is not allowe...

Hi @mohit7, There are two separate mechanisms at play here, and it helps to distinguish between them because log compaction itself does not remove your original commit JSON files. WHAT LOG COMPACTION DOES Log compaction creates additional .compacted....

@NW1000 Thanks for the thorough description. I can understand the concern about inconsistent behavior here. The previous reply from nayan_wylde is on the right track -- this is almost certainly a metadata caching issue, not data corruption. Let me gi...

@rahul_goyal This is a common point of confusion when setting up Snowflake federation in Databricks, so let me clarify what is happening. SHORT ANSWER The JDBC URL display showing "jdbc://" instead of "jdbc//" is expected behavior and is not a defect...

Hi @Sunny_singh, Welcome to Data Engineering! This is a great question and one that comes up frequently. There is no single "correct" answer because real-world sizing depends on data format, cloud provider, data skew, and more, but here is a solid fr...

@MarkV Appreciate you sharing the details. There are a few approaches depending on whether you want to use the REST API, the Python SDK, SQL, or the CLI. Here is a breakdown. BACKGROUND When you create a schedule for a query in the Databricks SQL edi...

@databricks_use2 -- Happy to help with this one. Moving from commercial to GovCloud with a full SDP medallion architecture requires careful planning. Here is a step-by-step approach that covers both the data migration and the pipeline rebuild. THE SH...

Hi @yashojha, Thanks for the detailed writeup. A 22-minute write for a small file to a managed volume is definitely not expected behavior, especially when it works quickly in your lower environment. Let me walk through what is likely happening and ho...