- 21 Views
- 2 replies
- 0 kudos
How to enable Liquid Clustering on an existing Delta Live Table (DLT) and syntax for enabling it
Hi all,I’m working with Delta Live Tables (DLT) and want to enable Liquid Clustering on an existing DLT table that was already created without it.Could someone please clarify:How can I enable Liquid Clustering on an existing DLT table (without recre...
- 21 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @Sakthi0311 ,For SQL language you can enable LC for materialized views and streaming tables. So the syntax looks following:If you want to use automatic clustering then use CLUSTER BY AUTO.
- 0 kudos
- 431 Views
- 6 replies
- 2 kudos
Resolved! How do use Databricks Lakeflow Declarative Pipeline on AWS DMS data?
Hi!I am trying to replicate an AWS RDS PostgreSQL database in Databricks. I have successfully manage to enable CDC using AWS DMS that writes an initial load file and continuous CDC files in parquet.I have been trying to follow the official guide Repl...
- 431 Views
- 6 replies
- 2 kudos
- 2 kudos
hey @excavator-matt Let's remember that the Bronze layer is for mere raw ingestion; this provides a baseline for auditing and to start applying transformations based on the different use cases you need to serve. Systems and their requirements change...
- 2 kudos
- 29 Views
- 4 replies
- 3 kudos
Resolved! Quota Limit Exhausted Error when Creating Data Ingestion with SQL Server Connector (Azure)
Good Day all,I am having an issue with our first Data Ingestion Pipelines, I am wanting to connect to our Azure SQL Server with our Unity Connector (I can access the data in Unity Catalog). When I am on Step 3 of the process (Source) when it is scann...
- 29 Views
- 4 replies
- 3 kudos
- 11118 Views
- 5 replies
- 3 kudos
Resolved! How does partition pruning work on a merge into statement?
I have a delta table that is partitioned by Year, Date and month. I'm trying to merge data to this on all three partition columns + an extra column (an ID). My merge statement is below:MERGE INTO delta.<path of delta table> oldData using df newData ...
- 11118 Views
- 5 replies
- 3 kudos
- 3 kudos
Isn't the suggested idea only filtering the input dataframe (resulting in a smaller amount of data to match across the whole delta table) rather than prune the delta table for relevant partitions to scan?
- 3 kudos
- 27 Views
- 3 replies
- 2 kudos
Does Autoloader supports loading PDF files?
I need to process PDF files already ingested. Based on the documentation, Autoloader does not support PDFs - or am I missing something?Also, I've found this sparkPDF library in other discussions in the community, but from what I see it's only for bat...
- 27 Views
- 3 replies
- 2 kudos
- 6746 Views
- 7 replies
- 0 kudos
How to Assign User Managed Identity to DBR Cluster so I can use it for quering ADLSv2?
Hi,I'm trying to figure out if we can switch from Entra ID SPN's to User Assigned Managed Indentities and everything works except I can't figure out how to access the lake files from python notebook.I've tried with below code and was running it on a ...
- 6746 Views
- 7 replies
- 0 kudos
- 0 kudos
Besides, this only works in dedicated clusters, non working on shared ones. Why? No idea at all. Latest case, IMDS (Internal Metadata Service) used by Azure to inject token endpoint inside resources as a unique secure and valid channel to get tokens ...
- 0 kudos
- 11 Views
- 0 replies
- 0 kudos
When to Use and when Not to Use Liquid Clustering?
Hi everyone,I’m looking for some practical guidance and experiences around when to choose Liquid Clustering versus sticking with traditional partitioning + Z-ordering.From what I’ve gathered so far:For small tables (<10TB), Liquid Clustering gives s...
- 11 Views
- 0 replies
- 0 kudos
- 2518 Views
- 1 replies
- 0 kudos
Genie Integrating with streamlit
Hi All,What are the best practices to follow while integrating with Genie and streamlit , and are there any limitations?how best way to present in UI level on user perceptive ?Regards,Phani
- 2518 Views
- 1 replies
- 0 kudos
- 0 kudos
This should help you get started. Please let us know if you've any specific question after you've looked at the links below. https://blog.streamlit.io/best-practices-for-building-genai-apps-with-streamlit/ https://databrickster.medium.com/call-genie-...
- 0 kudos
- 2262 Views
- 2 replies
- 1 kudos
Resolved! Syncing lakebase table to delta table
I have been exploring Lakebase and I wanted to know if there is a way to sync CDC data from Lakebase tables to delta table in Lakehouse. I know the other way is possible and that's what was shown in the demo. Can you tell how I can I sync both the ta...
- 2262 Views
- 2 replies
- 1 kudos
- 1 kudos
Just wanted to mention that the ETL from Lakebase to Delta Tables preview is mentioned here:https://www.databricks.com/blog/how-use-lakebase-transactional-data-layer-databricks-apps
- 1 kudos
- 13 Views
- 0 replies
- 0 kudos
Network error on subsequent runs using serverless compute in DLT
Hello,When running on a serverless cluster in DLT our notebook first tries to install some python whls onto the cluster. We have noticed that when in development and running a pipeline many times over in a short space of time between runs that the pi...
- 13 Views
- 0 replies
- 0 kudos
- 46 Views
- 3 replies
- 3 kudos
Hive Metastore End of Life
Hello everyone,is there a rough date on which the Hive Metastore will be deactivated?In the end, I ask the question again, which was already asked 2 years ago:Solved: Hive metastore table access control End of Support - Databricks Community - 50487We...
- 46 Views
- 3 replies
- 3 kudos
- 3 kudos
HI @julius_bkr I do not think Databricks has any plans to retire HMA completely. However, Unity Catalog is the strategic direction and the recommended approach for new deployments and migrating existing data governance. Databricks is investing heavil...
- 3 kudos
- 91 Views
- 4 replies
- 3 kudos
Resolved! Pass parameters between jobs
Hello I have jobIn that job, it runs a task (GetGid) that executes a notebook and obtains some value using dbutils.jobs.taskValuesSete.g. dbutils.jobs.taskValuesSet(key = "gid", value = gid)As a result, I can use this and pass it to another task for ...
- 91 Views
- 4 replies
- 3 kudos
- 3 kudos
Thanks @Hubert-Dudek and @ilir_nuredini I see this nowI'm setting using:dbutils.jobs.taskValues.Set()passing to the job task using Key - gid; Value - {{tasks.GetGid.values.gid}}Then reading using: pid = dbutils.widgets.get()
- 3 kudos
- 17 Views
- 0 replies
- 0 kudos
Broken s3 file paths in File Notifications for auto loader
Suddenly at "2025-10-23T14:12:48.409+00:00", coming file paths from file notification queue started to be urlencoded. Hence, our pipeline gets file not found exception. I think something has changed suddenly and broke notification system. Here are th...
- 17 Views
- 0 replies
- 0 kudos
- 4491 Views
- 2 replies
- 1 kudos
Delta live tables - foreign keys
I'm creating ingestions using delta live tables, the dlt support the use of schema, with constraints like foreign keys. The problem is: how can i create foreign keys between the same pipeline, that has no read/write relation, but has foreign key rela...
- 4491 Views
- 2 replies
- 1 kudos
- 1 kudos
We ran into similar issues with our client, while migrating their on-prem table relationships to Databricks via DLT/LDP.The first proposed solution does not work on Materialized Views (MV).> 'ALTER TABLE ... ADD CONSTRAINT' expects a table but <objec...
- 1 kudos
- 31 Views
- 0 replies
- 0 kudos
Inquiry on GraphFrame Library Upgrade Timeline for Databricks Runtime for Machine Learning
Thanks for the Databricks community and maintaining such a valuable platform.I would like to inquire if there is a planned timeline for upgrading the GraphFrame library. We’ve noticed that the latest release on GitHub is v0.9.3, while the Databricks ...
- 31 Views
- 0 replies
- 0 kudos
| User | Count |
|---|---|
| 1619 | |
| 790 | |
| 473 | |
| 349 | |
| 287 |
