- 113 Views
- 4 replies
- 1 kudos
Databricks Python stored procedures
Hi,I am using databricks runtime 17.3.x-scala2.13 , But I am unable to create python stored procedures, (functions are possible but they dont support a spark session like below) , any thoughts/help is much appreciated ? [INVALID_STATEMENT_OR_CLAUSE]...
- 113 Views
- 4 replies
- 1 kudos
- 1 kudos
Hi As others have said stored procedures don't currently support Python. You can either create the stored procedure with SQL using Windows functions and describe history or put it into a notebook and not have stored procedures. The video is about exp...
- 1 kudos
- 53 Views
- 2 replies
- 1 kudos
Find and replace not saving changes in Databricks Notebook problem
In Databricks notebooks, the find and replace functionality doesn't appear to be saving the replaced text. When I use find and replace text, it does initially appear to work correctly (I see that the highlighted texts are replaced).However, when I sc...
- 53 Views
- 2 replies
- 1 kudos
- 1 kudos
@milan2 Not really - been using Notebooks actively for past couple of days. Been using the Find & Replace quite often to replace empty lines/comments and it works absolutely fine.
- 1 kudos
- 92 Views
- 4 replies
- 2 kudos
SQL Warehouse stuck on "Cluster Start-up Delayed"
Hi everyone,I'm running into an issue with my Starter Warehouse on Databricks and would appreciate any help or pointers.Problem: My SQL Warehouse has been stuck in a Starting state with the following warning:Cluster Start-up Delayed. Please wait whil...
- 92 Views
- 4 replies
- 2 kudos
- 2 kudos
Hi @Nid-cbs, Sorry you're running into this. A few minutes of startup time can occur while the warehouse is provisioning, but if it has been stuck in Starting for quite a while, it could also be related to cloud quota or backend capacity, so it would...
- 2 kudos
- 3708 Views
- 3 replies
- 0 kudos
Tutorbees logo black
We provide online tutoring for students from Grade 5 and all the way up to professionals. You can find the best tutors for Maths, Biology, Physics, Chemistry, English, Social Sciences, Urdu in the comfort of your home. You can also find professional ...
- 3708 Views
- 3 replies
- 0 kudos
- 0 kudos
Along with Online tutoring, it offers in-demand courses like digital marketing, and online earning skills. Students also get access to free notes, study materials, and learning resources to support their academic journey. More than just an academy, B...
- 0 kudos
- 268 Views
- 4 replies
- 0 kudos
Liquid Clustering VS Z-ordering
I want to understand difference b/w Liquid Clustering VS Z-ordering and also how both works?
- 268 Views
- 4 replies
- 0 kudos
- 0 kudos
Z-Ordering physically sorts data using multi-dimensional ordering, but degrades as new data arrives — requiring full, expensive OPTIMIZE reruns to maintain.Liquid Clustering (DBR 13.3+) replaces both Z-Ordering and Hive partitioning. You define it on...
- 0 kudos
- 1154 Views
- 4 replies
- 2 kudos
Resolved! Not able to find lab for Data Engineering Learning Path
I am not able to find the data engineering learning path , i did open partner databricks academy lab which redirected to uplimit where i also enrolled myself to instructor led course but not able to see any labs.
- 1154 Views
- 4 replies
- 2 kudos
- 2 kudos
Thanks, Junaid! This cleared things up for me. I’ve enrolled and received the session invites, so I’ll wait for the next cohort to start and check for lab access then
- 2 kudos
- 171 Views
- 3 replies
- 0 kudos
Enable CDC in Lakeflow Connect Tables
Hello everyone, I'm implementing a project that requires AutoCDC workflows using Lakeflow.The architecture is as follows: Data is ingested from a database using Lakeflow Connect, and then a declarative pipeline performs some transformations on this d...
- 171 Views
- 3 replies
- 0 kudos
- 0 kudos
Lakeflow Connect tables generally already include the CDC metadata required for downstream processing. I'd first check:SHOW TBLPROPERTIES <catalog>.<schema>.<table>; If delta.enableChangeDataFeed=true is present, you can directly consume the table w...
- 0 kudos
- 2607 Views
- 1 replies
- 2 kudos
java.lang.SecurityException: Could not verify permissions for OverwritePartitionsDynamic RelationV2 - Delta tables dynamic partition overwrite on Databricks ACL enabled clusters
I'm working on Databricks ACL enabled clusters, and having trouble performing dynamic partition overwrite to Delta tables.I have created a test table using the following query:CREATE TABLE IF NOT EXISTS test_01 ( id STRING, name STRING, c...
- 2607 Views
- 1 replies
- 2 kudos
- 2 kudos
This behaviour is expected on some Databricks Runtime versions when Table ACLs (ACL-enabled clusters) are enabled.The key indicator is the error:java.lang.SecurityException: Could not verify permissions for OverwritePartitionsDynamic RelationV2Your o...
- 2 kudos
- 198 Views
- 2 replies
- 0 kudos
DLT pipeline's compute policy when Instance pool Id used it ignores the VM series.
Hi In Lake flow Spark Declarative Pipelines (formerly DLT) I’m trying to understand how instance pool, cluster policy and DLT pipeline interact, especially around instance type selection.I created an instance pool with Instance type: Standard_DS3_v2 ...
- 198 Views
- 2 replies
- 0 kudos
- 0 kudos
I tried to reproduce and it worked as expected for me. Check your pipeline JSON to make sure your "clusters" spec there is explicitly mentioning the pool details. Example: "pipeline_type": "WORKSPACE", "name": "mojama-dlt-classic-demo", "cluste...
- 0 kudos
- 124 Views
- 2 replies
- 0 kudos
PostgreSQL ingestion source not supported in workspace when deploying Databricks Asset Bundle
I'm trying to deploy a Databricks Asset Bundle that creates a DLT/Lakeflow pipeline with a PostgreSQL ingestion source. The bundle builds successfully, uploads artifacts, and starts deploying resources, but the pipeline creation fails with the follow...
- 124 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @muaaz ,Yep, that feature is in public preview. Usually, when something is in public preview you have access to that feature, but in this case they require some steps to enroll it for a given workspace. So, as @balajij8 and docs suggests -> conta...
- 0 kudos
- 275 Views
- 6 replies
- 1 kudos
Resolved! Automate Lakeflow connect to ingest 300 tables not manually
I have data in PostgreSQL and I’m using Lakeflow Connect via UI to ingest it into Databricks streaming tables.Currently, each Lakeflow Connect pipeline only allows connecting one PostgreSQL table. I have around 300 tables, and creating pipelines manu...
- 275 Views
- 6 replies
- 1 kudos
- 87 Views
- 0 replies
- 0 kudos
Apache Spark Masterclass (In-Person, Bengaluru) | 6 June
Hi everyone,We're hosting the next session of our Data Engineering Masterclass Series focused on Apache Spark.This is an in-person, hands-on session for engineers interested in modern data engineering, distributed data processing, and real-world Spar...
- 87 Views
- 0 replies
- 0 kudos
- 281 Views
- 3 replies
- 0 kudos
Resolved! Databricks SQL connection becomes stale in long-running app
I’m building a Databricks App that continuously queries a SQL Warehouse roughly every 30 seconds to retrieve updated data.To avoid the overhead of repeatedly opening new connections, I’m currently caching the Databricks SQL connection using lru_cache...
- 281 Views
- 3 replies
- 0 kudos
- 0 kudos
SQLAlchemy dialect is a wrapper for the native databricks sql connector. You can try to pass the various authentication configuration supported by the underlying SQL connector directly into the connect_args dictionary parameter of the alchemy engine....
- 0 kudos
- 3414 Views
- 8 replies
- 1 kudos
Resolved! Delta Live Tables - skipChangeCommits in SQL
Hi,Could anyone tell me if the skipChangeCommits option is supported in SQL mode? I can use it successfully using Python, but it doesn't look like it is supported by SQL.It seems to be a glaring omission from the SQL support, or support for this will...
- 3414 Views
- 8 replies
- 1 kudos
- 1 kudos
This is now supported:CREATE OR REFRESH STREAMING TABLE basic_stAS SELECT * FROM STREAM samples.nyctaxi.trips WITH (SKIPCHANGECOMMITS);Supported in runtime 17.3 and later.Documentation: https://docs.databricks.com/aws/en/ldp/developer/sql-dev#create-...
- 1 kudos
- 94 Views
- 0 replies
- 0 kudos
Why We Used Two Bronze Tables Instead of One — And Why It Mattered
Post removed — reposting with corrections
- 94 Views
- 0 replies
- 0 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 568 | |
| 349 | |
| 287 |
