Hi everyone,I’m looking for some practical guidance and experiences around when to choose Liquid Clustering versus sticking with traditional partitioning + Z-ordering.From what I’ve gathered so far:For small tables (<10TB), Liquid Clustering gives s...
Good Day all,I am having an issue with our first Data Ingestion Pipelines, I am wanting to connect to our Azure SQL Server with our Unity Connector (I can access the data in Unity Catalog). When I am on Step 3 of the process (Source) when it is scann...
Hi @Adam_Borlase ,I've checked and by default if you don't configure gateway setting yourself it will create VM with following types. So check if you didn't exceed quotas for Standard_F4s or for Standard_E4d_v4 family of VMs.My finding is in line wit...
Hi All,What are the best practices to follow while integrating with Genie and streamlit , and are there any limitations?how best way to present in UI level on user perceptive ?Regards,Phani
This should help you get started. Please let us know if you've any specific question after you've looked at the links below. https://blog.streamlit.io/best-practices-for-building-genai-apps-with-streamlit/ https://databrickster.medium.com/call-genie-...
I have been exploring Lakebase and I wanted to know if there is a way to sync CDC data from Lakebase tables to delta table in Lakehouse. I know the other way is possible and that's what was shown in the demo. Can you tell how I can I sync both the ta...
Just wanted to mention that the ETL from Lakebase to Delta Tables preview is mentioned here:https://www.databricks.com/blog/how-use-lakebase-transactional-data-layer-databricks-apps
Hello,When running on a serverless cluster in DLT our notebook first tries to install some python whls onto the cluster. We have noticed that when in development and running a pipeline many times over in a short space of time between runs that the pi...
Hello everyone,is there a rough date on which the Hive Metastore will be deactivated?In the end, I ask the question again, which was already asked 2 years ago:Solved: Hive metastore table access control End of Support - Databricks Community - 50487We...
HI @julius_bkr I do not think Databricks has any plans to retire HMA completely. However, Unity Catalog is the strategic direction and the recommended approach for new deployments and migrating existing data governance. Databricks is investing heavil...
Hello I have jobIn that job, it runs a task (GetGid) that executes a notebook and obtains some value using dbutils.jobs.taskValuesSete.g. dbutils.jobs.taskValuesSet(key = "gid", value = gid)As a result, I can use this and pass it to another task for ...
Thanks @Hubert-Dudek and @ilir_nuredini I see this nowI'm setting using:dbutils.jobs.taskValues.Set()passing to the job task using Key - gid; Value - {{tasks.GetGid.values.gid}}Then reading using: pid = dbutils.widgets.get()
Suddenly at "2025-10-23T14:12:48.409+00:00", coming file paths from file notification queue started to be urlencoded. Hence, our pipeline gets file not found exception. I think something has changed suddenly and broke notification system. Here are th...
I'm creating ingestions using delta live tables, the dlt support the use of schema, with constraints like foreign keys. The problem is: how can i create foreign keys between the same pipeline, that has no read/write relation, but has foreign key rela...
We ran into similar issues with our client, while migrating their on-prem table relationships to Databricks via DLT/LDP.The first proposed solution does not work on Materialized Views (MV).> 'ALTER TABLE ... ADD CONSTRAINT' expects a table but <objec...
Thanks for the Databricks community and maintaining such a valuable platform.I would like to inquire if there is a planned timeline for upgrading the GraphFrame library. We’ve noticed that the latest release on GitHub is v0.9.3, while the Databricks ...
I am using dataricks version 15.4 and getting below error whicle reading from jdbc and writing to aws S3 location: [NUMERIC_VALUE_OUT_OF_RANGE.WITHOUT_SUGGESTION] The -12874815911431.6200000000 rounded half up from -12874815911431.6200000000 cannot b...
After years of juggling tasks and feeling constantly behind, I tried something different: time-blocking. Instead of reacting to everything at once, I now assign focused time slots to specific tasks — and the difference is night and day. My productivi...
I'm looking for SQL linters in two different contexts within Databricks GUI:SQL queries in Databricks SQL Editor (GUI) - Is there a built-in linter for writing SQL queries in the Databricks SQL workspace?%sql magic cells in Databricks notebooks (GUI...
in databricks notebook is possible to connect own linters but I don't think it will work in SQL editor.
I am getting a pop-up error message in the right hand corner of my Databricks session, "Your notebook session has crashed." This is a notebook I've been working in for a while.The only line of code I am running is pip listThe error at the bottom of ...
From the screen, it looks like an interactive cluster—maybe try rebooting it.
I'm looking for a SQL formatter (CLI) for two different contexts within Databricks.1. SQL queries in .sql files - What tool does Databricks use by default to format SQL?2. %sql magic cells in notebooks (.ipynb files) - What tool does Databricks use b...
Hi @Lenoq ,As far as I know, currently there is no tool that meets your given criteria by databricks/labs. The closest that I can see is the tool you gave as an example, but as you said its not that good. what I would do maybe is an intermedial optio...
| User | Count |
|---|---|
| 1619 | |
| 790 | |
| 473 | |
| 349 | |
| 287 |
