- 1284 Views
- 2 replies
- 0 kudos
Lakeflow Connect - Column filtering
Hi community , I am interested in learning more about the feature that was mentioned in recent summit about query pushdown in lakeflow connect for SQL server. I believe this feature will allow to select only the required columns from source tables. I...
- 1284 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @amitpm, Just closing the loop on this thread, and apologies for the very late follow-up. The feature being discussed here maps to Lakeflow Connect query-based connectors for SQL Server, and that capability is now available. Query-based connectors...
- 0 kudos
- 499 Views
- 2 replies
- 1 kudos
Resolved! MongoDB Spark Connector UC command is not supported without recommendation
I am trying to read a MongoDB collection using spark.read.format("mongodb"). However, when I attempt to display the collection, I receive the error: "UC command is not supported without recommendation." Please help resolve this issue.
- 499 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @shan-databricks, This is expected behaviour on Unity Catalog shared or standard compute. The mongodb Spark data source path can fail there with UC_COMMAND_NOT_SUPPORTED.WITHOUT_RECOMMENDATION / SQLSTATE 0AKUC. The public docs for that are UC_COMM...
- 1 kudos
- 2971 Views
- 2 replies
- 4 kudos
Delta Sharing Costs
When Delta Sharing is enabled and a link is shared, I understand that the data transfer happens directly and not through the sharing server. I'm curious how costs are calculated. Is the entity making the share available charged for data egress and ...
- 2971 Views
- 2 replies
- 4 kudos
- 4 kudos
I am also curious about this - is there anyway to monitor this apart from system.billing.usage table?
- 4 kudos
- 771 Views
- 3 replies
- 1 kudos
Resolved! Find and replace not saving changes in Databricks Notebook problem
In Databricks notebooks, the find and replace functionality doesn't appear to be saving the replaced text. When I use find and replace text, it does initially appear to work correctly (I see that the highlighted texts are replaced).However, when I sc...
- 771 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi @milan2, Thanks for raising this. What you are describing sounds like the text is being visually replaced in the editor, but the change is not actually being persisted. We have also seen a recent report of similar behaviour where using the browser...
- 1 kudos
- 1052 Views
- 4 replies
- 6 kudos
Resolved! Databricks Python stored procedures
Hi,I am using databricks runtime 17.3.x-scala2.13 , But I am unable to create python stored procedures, (functions are possible but they dont support a spark session like below) , any thoughts/help is much appreciated ? [INVALID_STATEMENT_OR_CLAUSE]...
- 1052 Views
- 4 replies
- 6 kudos
- 6 kudos
Hi As others have said stored procedures don't currently support Python. You can either create the stored procedure with SQL using Windows functions and describe history or put it into a notebook and not have stored procedures. The video is about exp...
- 6 kudos
- 1173 Views
- 4 replies
- 4 kudos
Resolved! SQL Warehouse stuck on "Cluster Start-up Delayed"
Hi everyone,I'm running into an issue with my Starter Warehouse on Databricks and would appreciate any help or pointers.Problem: My SQL Warehouse has been stuck in a Starting state with the following warning:Cluster Start-up Delayed. Please wait whil...
- 1173 Views
- 4 replies
- 4 kudos
- 4 kudos
Hi @Nid-cbs, Sorry you're running into this. A few minutes of startup time can occur while the warehouse is provisioning, but if it has been stuck in Starting for quite a while, it could also be related to cloud quota or backend capacity, so it would...
- 4 kudos
- 1001 Views
- 4 replies
- 0 kudos
Liquid Clustering VS Z-ordering
I want to understand difference b/w Liquid Clustering VS Z-ordering and also how both works?
- 1001 Views
- 4 replies
- 0 kudos
- 0 kudos
Z-Ordering physically sorts data using multi-dimensional ordering, but degrades as new data arrives — requiring full, expensive OPTIMIZE reruns to maintain.Liquid Clustering (DBR 13.3+) replaces both Z-Ordering and Hive partitioning. You define it on...
- 0 kudos
- 1688 Views
- 4 replies
- 2 kudos
Resolved! Not able to find lab for Data Engineering Learning Path
I am not able to find the data engineering learning path , i did open partner databricks academy lab which redirected to uplimit where i also enrolled myself to instructor led course but not able to see any labs.
- 1688 Views
- 4 replies
- 2 kudos
- 2 kudos
Thanks, Junaid! This cleared things up for me. I’ve enrolled and received the session invites, so I’ll wait for the next cohort to start and check for lab access then
- 2 kudos
- 789 Views
- 3 replies
- 0 kudos
Resolved! Enable CDC in Lakeflow Connect Tables
Hello everyone, I'm implementing a project that requires AutoCDC workflows using Lakeflow.The architecture is as follows: Data is ingested from a database using Lakeflow Connect, and then a declarative pipeline performs some transformations on this d...
- 789 Views
- 3 replies
- 0 kudos
- 0 kudos
Lakeflow Connect tables generally already include the CDC metadata required for downstream processing. I'd first check:SHOW TBLPROPERTIES <catalog>.<schema>.<table>; If delta.enableChangeDataFeed=true is present, you can directly consume the table w...
- 0 kudos
- 2873 Views
- 1 replies
- 2 kudos
java.lang.SecurityException: Could not verify permissions for OverwritePartitionsDynamic RelationV2 - Delta tables dynamic partition overwrite on Databricks ACL enabled clusters
I'm working on Databricks ACL enabled clusters, and having trouble performing dynamic partition overwrite to Delta tables.I have created a test table using the following query:CREATE TABLE IF NOT EXISTS test_01 ( id STRING, name STRING, c...
- 2873 Views
- 1 replies
- 2 kudos
- 2 kudos
This behaviour is expected on some Databricks Runtime versions when Table ACLs (ACL-enabled clusters) are enabled.The key indicator is the error:java.lang.SecurityException: Could not verify permissions for OverwritePartitionsDynamic RelationV2Your o...
- 2 kudos
- 529 Views
- 2 replies
- 0 kudos
DLT pipeline's compute policy when Instance pool Id used it ignores the VM series.
Hi In Lake flow Spark Declarative Pipelines (formerly DLT) I’m trying to understand how instance pool, cluster policy and DLT pipeline interact, especially around instance type selection.I created an instance pool with Instance type: Standard_DS3_v2 ...
- 529 Views
- 2 replies
- 0 kudos
- 0 kudos
I tried to reproduce and it worked as expected for me. Check your pipeline JSON to make sure your "clusters" spec there is explicitly mentioning the pool details. Example: "pipeline_type": "WORKSPACE", "name": "mojama-dlt-classic-demo", "cluste...
- 0 kudos
- 537 Views
- 2 replies
- 1 kudos
Resolved! PostgreSQL ingestion source not supported in workspace when deploying Databricks Asset Bundle
I'm trying to deploy a Databricks Asset Bundle that creates a DLT/Lakeflow pipeline with a PostgreSQL ingestion source. The bundle builds successfully, uploads artifacts, and starts deploying resources, but the pipeline creation fails with the follow...
- 537 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @muaaz ,Yep, that feature is in public preview. Usually, when something is in public preview you have access to that feature, but in this case they require some steps to enroll it for a given workspace. So, as @balajij8 and docs suggests -> conta...
- 1 kudos
- 1319 Views
- 6 replies
- 1 kudos
Resolved! Automate Lakeflow connect to ingest 300 tables not manually
I have data in PostgreSQL and I’m using Lakeflow Connect via UI to ingest it into Databricks streaming tables.Currently, each Lakeflow Connect pipeline only allows connecting one PostgreSQL table. I have around 300 tables, and creating pipelines manu...
- 1319 Views
- 6 replies
- 1 kudos
- 946 Views
- 3 replies
- 0 kudos
Resolved! Databricks SQL connection becomes stale in long-running app
I’m building a Databricks App that continuously queries a SQL Warehouse roughly every 30 seconds to retrieve updated data.To avoid the overhead of repeatedly opening new connections, I’m currently caching the Databricks SQL connection using lru_cache...
- 946 Views
- 3 replies
- 0 kudos
- 0 kudos
SQLAlchemy dialect is a wrapper for the native databricks sql connector. You can try to pass the various authentication configuration supported by the underlying SQL connector directly into the connect_args dictionary parameter of the alchemy engine....
- 0 kudos
- 4139 Views
- 8 replies
- 1 kudos
Resolved! Delta Live Tables - skipChangeCommits in SQL
Hi,Could anyone tell me if the skipChangeCommits option is supported in SQL mode? I can use it successfully using Python, but it doesn't look like it is supported by SQL.It seems to be a glaring omission from the SQL support, or support for this will...
- 4139 Views
- 8 replies
- 1 kudos
- 1 kudos
This is now supported:CREATE OR REFRESH STREAMING TABLE basic_stAS SELECT * FROM STREAM samples.nyctaxi.trips WITH (SKIPCHANGECOMMITS);Supported in runtime 17.3 and later.Documentation: https://docs.databricks.com/aws/en/ldp/developer/sql-dev#create-...
- 1 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 581 | |
| 349 | |
| 287 |
