- 86 Views
- 4 replies
- 5 kudos
Resolved! Quota Limit Exhausted Error when Creating Data Ingestion with SQL Server Connector (Azure)
Good Day all,I am having an issue with our first Data Ingestion Pipelines, I am wanting to connect to our Azure SQL Server with our Unity Connector (I can access the data in Unity Catalog). When I am on Step 3 of the process (Source) when it is scann...
- 86 Views
- 4 replies
- 5 kudos
- 11169 Views
- 5 replies
- 3 kudos
Resolved! How does partition pruning work on a merge into statement?
I have a delta table that is partitioned by Year, Date and month. I'm trying to merge data to this on all three partition columns + an extra column (an ID). My merge statement is below:MERGE INTO delta.<path of delta table> oldData using df newData ...
- 11169 Views
- 5 replies
- 3 kudos
- 3 kudos
Isn't the suggested idea only filtering the input dataframe (resulting in a smaller amount of data to match across the whole delta table) rather than prune the delta table for relevant partitions to scan?
- 3 kudos
- 60 Views
- 3 replies
- 3 kudos
Does Autoloader supports loading PDF files?
I need to process PDF files already ingested. Based on the documentation, Autoloader does not support PDFs - or am I missing something?Also, I've found this sparkPDF library in other discussions in the community, but from what I see it's only for bat...
- 60 Views
- 3 replies
- 3 kudos
- 6780 Views
- 7 replies
- 0 kudos
How to Assign User Managed Identity to DBR Cluster so I can use it for quering ADLSv2?
Hi,I'm trying to figure out if we can switch from Entra ID SPN's to User Assigned Managed Indentities and everything works except I can't figure out how to access the lake files from python notebook.I've tried with below code and was running it on a ...
- 6780 Views
- 7 replies
- 0 kudos
- 0 kudos
Besides, this only works in dedicated clusters, non working on shared ones. Why? No idea at all. Latest case, IMDS (Internal Metadata Service) used by Azure to inject token endpoint inside resources as a unique secure and valid channel to get tokens ...
- 0 kudos
- 2556 Views
- 1 replies
- 0 kudos
Genie Integrating with streamlit
Hi All,What are the best practices to follow while integrating with Genie and streamlit , and are there any limitations?how best way to present in UI level on user perceptive ?Regards,Phani
- 2556 Views
- 1 replies
- 0 kudos
- 0 kudos
This should help you get started. Please let us know if you've any specific question after you've looked at the links below. https://blog.streamlit.io/best-practices-for-building-genai-apps-with-streamlit/ https://databrickster.medium.com/call-genie-...
- 0 kudos
- 2287 Views
- 2 replies
- 1 kudos
Resolved! Syncing lakebase table to delta table
I have been exploring Lakebase and I wanted to know if there is a way to sync CDC data from Lakebase tables to delta table in Lakehouse. I know the other way is possible and that's what was shown in the demo. Can you tell how I can I sync both the ta...
- 2287 Views
- 2 replies
- 1 kudos
- 1 kudos
Just wanted to mention that the ETL from Lakebase to Delta Tables preview is mentioned here:https://www.databricks.com/blog/how-use-lakebase-transactional-data-layer-databricks-apps
- 1 kudos
- 42 Views
- 0 replies
- 0 kudos
Network error on subsequent runs using serverless compute in DLT
Hello,When running on a serverless cluster in DLT our notebook first tries to install some python whls onto the cluster. We have noticed that when in development and running a pipeline many times over in a short space of time between runs that the pi...
- 42 Views
- 0 replies
- 0 kudos
- 75 Views
- 3 replies
- 3 kudos
Hive Metastore End of Life
Hello everyone,is there a rough date on which the Hive Metastore will be deactivated?In the end, I ask the question again, which was already asked 2 years ago:Solved: Hive metastore table access control End of Support - Databricks Community - 50487We...
- 75 Views
- 3 replies
- 3 kudos
- 3 kudos
HI @julius_bkr I do not think Databricks has any plans to retire HMA completely. However, Unity Catalog is the strategic direction and the recommended approach for new deployments and migrating existing data governance. Databricks is investing heavil...
- 3 kudos
- 34 Views
- 0 replies
- 0 kudos
Broken s3 file paths in File Notifications for auto loader
Suddenly at "2025-10-23T14:12:48.409+00:00", coming file paths from file notification queue started to be urlencoded. Hence, our pipeline gets file not found exception. I think something has changed suddenly and broke notification system. Here are th...
- 34 Views
- 0 replies
- 0 kudos
- 4531 Views
- 2 replies
- 1 kudos
Delta live tables - foreign keys
I'm creating ingestions using delta live tables, the dlt support the use of schema, with constraints like foreign keys. The problem is: how can i create foreign keys between the same pipeline, that has no read/write relation, but has foreign key rela...
- 4531 Views
- 2 replies
- 1 kudos
- 1 kudos
We ran into similar issues with our client, while migrating their on-prem table relationships to Databricks via DLT/LDP.The first proposed solution does not work on Materialized Views (MV).> 'ALTER TABLE ... ADD CONSTRAINT' expects a table but <objec...
- 1 kudos
- 47 Views
- 0 replies
- 0 kudos
Inquiry on GraphFrame Library Upgrade Timeline for Databricks Runtime for Machine Learning
Thanks for the Databricks community and maintaining such a valuable platform.I would like to inquire if there is a planned timeline for upgrading the GraphFrame library. We’ve noticed that the latest release on GitHub is v0.9.3, while the Databricks ...
- 47 Views
- 0 replies
- 0 kudos
- 144 Views
- 6 replies
- 3 kudos
Resolved! [NUMERIC_VALUE_OUT_OF_RANGE.WITHOUT_SUGGESTION] The -12874815911431.6200000000 rounded half up from
I am using dataricks version 15.4 and getting below error whicle reading from jdbc and writing to aws S3 location: [NUMERIC_VALUE_OUT_OF_RANGE.WITHOUT_SUGGESTION] The -12874815911431.6200000000 rounded half up from -12874815911431.6200000000 cannot b...
- 144 Views
- 6 replies
- 3 kudos
- 141 Views
- 4 replies
- 2 kudos
Are there SQL linters for Databricks GUI queries and notebook %sql cells?
I'm looking for SQL linters in two different contexts within Databricks GUI:SQL queries in Databricks SQL Editor (GUI) - Is there a built-in linter for writing SQL queries in the Databricks SQL workspace?%sql magic cells in Databricks notebooks (GUI...
- 141 Views
- 4 replies
- 2 kudos
- 2 kudos
in databricks notebook is possible to connect own linters but I don't think it will work in SQL editor.
- 2 kudos
- 180 Views
- 4 replies
- 1 kudos
Resolved! Notebook Session Has Crashed
I am getting a pop-up error message in the right hand corner of my Databricks session, "Your notebook session has crashed." This is a notebook I've been working in for a while.The only line of code I am running is pip listThe error at the bottom of ...
- 180 Views
- 4 replies
- 1 kudos
- 1 kudos
From the screen, it looks like an interactive cluster—maybe try rebooting it.
- 1 kudos
- 81 Views
- 1 replies
- 0 kudos
Looking for CLI-based SQL formatter for Databricks: Alternative to gethue/sql-formatter for .
I'm looking for a SQL formatter (CLI) for two different contexts within Databricks.1. SQL queries in .sql files - What tool does Databricks use by default to format SQL?2. %sql magic cells in notebooks (.ipynb files) - What tool does Databricks use b...
- 81 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Lenoq ,As far as I know, currently there is no tool that meets your given criteria by databricks/labs. The closest that I can see is the tool you gave as an example, but as you said its not that good. what I would do maybe is an intermedial optio...
- 0 kudos
| User | Count |
|---|---|
| 1619 | |
| 790 | |
| 480 | |
| 349 | |
| 287 |
