- 2919 Views
- 1 replies
- 2 kudos
java.lang.SecurityException: Could not verify permissions for OverwritePartitionsDynamic RelationV2 - Delta tables dynamic partition overwrite on Databricks ACL enabled clusters
I'm working on Databricks ACL enabled clusters, and having trouble performing dynamic partition overwrite to Delta tables.I have created a test table using the following query:CREATE TABLE IF NOT EXISTS test_01 ( id STRING, name STRING, c...
- 2919 Views
- 1 replies
- 2 kudos
- 2 kudos
This behaviour is expected on some Databricks Runtime versions when Table ACLs (ACL-enabled clusters) are enabled.The key indicator is the error:java.lang.SecurityException: Could not verify permissions for OverwritePartitionsDynamic RelationV2Your o...
- 2 kudos
- 617 Views
- 2 replies
- 0 kudos
DLT pipeline's compute policy when Instance pool Id used it ignores the VM series.
Hi In Lake flow Spark Declarative Pipelines (formerly DLT) I’m trying to understand how instance pool, cluster policy and DLT pipeline interact, especially around instance type selection.I created an instance pool with Instance type: Standard_DS3_v2 ...
- 617 Views
- 2 replies
- 0 kudos
- 0 kudos
I tried to reproduce and it worked as expected for me. Check your pipeline JSON to make sure your "clusters" spec there is explicitly mentioning the pool details. Example: "pipeline_type": "WORKSPACE", "name": "mojama-dlt-classic-demo", "cluste...
- 0 kudos
- 641 Views
- 2 replies
- 1 kudos
Resolved! PostgreSQL ingestion source not supported in workspace when deploying Databricks Asset Bundle
I'm trying to deploy a Databricks Asset Bundle that creates a DLT/Lakeflow pipeline with a PostgreSQL ingestion source. The bundle builds successfully, uploads artifacts, and starts deploying resources, but the pipeline creation fails with the follow...
- 641 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @muaaz ,Yep, that feature is in public preview. Usually, when something is in public preview you have access to that feature, but in this case they require some steps to enroll it for a given workspace. So, as @balajij8 and docs suggests -> conta...
- 1 kudos
- 1505 Views
- 6 replies
- 1 kudos
Resolved! Automate Lakeflow connect to ingest 300 tables not manually
I have data in PostgreSQL and I’m using Lakeflow Connect via UI to ingest it into Databricks streaming tables.Currently, each Lakeflow Connect pipeline only allows connecting one PostgreSQL table. I have around 300 tables, and creating pipelines manu...
- 1505 Views
- 6 replies
- 1 kudos
- 1101 Views
- 3 replies
- 0 kudos
Resolved! Databricks SQL connection becomes stale in long-running app
I’m building a Databricks App that continuously queries a SQL Warehouse roughly every 30 seconds to retrieve updated data.To avoid the overhead of repeatedly opening new connections, I’m currently caching the Databricks SQL connection using lru_cache...
- 1101 Views
- 3 replies
- 0 kudos
- 0 kudos
SQLAlchemy dialect is a wrapper for the native databricks sql connector. You can try to pass the various authentication configuration supported by the underlying SQL connector directly into the connect_args dictionary parameter of the alchemy engine....
- 0 kudos
- 4316 Views
- 8 replies
- 1 kudos
Resolved! Delta Live Tables - skipChangeCommits in SQL
Hi,Could anyone tell me if the skipChangeCommits option is supported in SQL mode? I can use it successfully using Python, but it doesn't look like it is supported by SQL.It seems to be a glaring omission from the SQL support, or support for this will...
- 4316 Views
- 8 replies
- 1 kudos
- 1 kudos
This is now supported:CREATE OR REFRESH STREAMING TABLE basic_stAS SELECT * FROM STREAM samples.nyctaxi.trips WITH (SKIPCHANGECOMMITS);Supported in runtime 17.3 and later.Documentation: https://docs.databricks.com/aws/en/ldp/developer/sql-dev#create-...
- 1 kudos
- 220 Views
- 0 replies
- 0 kudos
Why We Used Two Bronze Tables Instead of One — And Why It Mattered
Post removed — reposting with corrections
- 220 Views
- 0 replies
- 0 kudos
- 728 Views
- 4 replies
- 0 kudos
Renaming a folder in adls is taking a lot of time
Hi i have a folder customer and customer_01 in adls location , now i need to rename customer_01 to customer and customer to customer_01 both if these folder have lots of files . If i use dbutls.fs.mv its taking a lot of time like 7 hours something is...
- 728 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi @Nkrom ,I am happy to share the Azure REST API method! Using the Azure Python SDK is the absolute fastest way to do this but you can choose any other programming language.ADLS Gen2 uses a "Hierarchical Namespace" (HNS). When you use the Azure SDK ...
- 0 kudos
- 1307 Views
- 3 replies
- 3 kudos
Resolved! Missing upstream column lineage missing from api call after some time
I ran the following piece of code on 2 occasions.table_name = 'full path of table"lineage = w.api_client.do("GET",f"/api/2.0/lineage-tracking/column-lineage",body={"table_name": table_name,"column_name": "column_x"})u_lineage_df = spark.createDataFra...
- 1307 Views
- 3 replies
- 3 kudos
- 3 kudos
Hi @Mario_D, From what I can gather, this can happen, and it’s usually less about a restriction on calling the API itself and more about how lineage was captured or what the caller is allowed to see. A few common reasons are: The caller no longer has...
- 3 kudos
- 3092 Views
- 7 replies
- 9 kudos
Resolved! disabling a task in a databricks job
Hello I have jobs that perform a number of activities.Sometimes I want to disable one or more of these activities.I can do that easily in an app like ADF.Reading around I cannot find an easy way to do this although, what I've read suggest that it was...
- 3092 Views
- 7 replies
- 9 kudos
- 9 kudos
Hi folks, this is Gabor from Databricks here!We've recently launched Disabled tasks which lets you disable an task indefinitely so that it won't run the next time the job runs on a schedule. Simply specify "disabled": true in the job's configuration ...
- 9 kudos
- 3674 Views
- 4 replies
- 2 kudos
Resolved! Autoloader cloudFiles.maxFilesPerTrigger ignored with .trigger(availableNow=True)?
Hi, I'm using the Auto Loader feature to read streaming data from Delta Lake files and process them in a batch. The trigger is set to availableNow to include all new data from the checkpoint offset but I limit the amount of delta files for the batch ...
- 3674 Views
- 4 replies
- 2 kudos
- 2 kudos
Hi @johschmidt42 ,This is a great question, but the mystery actually lies in the very first line of your read configuration: spark_session.readStream.format(source="delta")Because you are using .format("delta") instead of .format("cloudFiles"), you a...
- 2 kudos
- 693 Views
- 2 replies
- 0 kudos
Resolved! Lakeflow SDP equivalent of whenNotMatchedBySource
I have a Lakeflow Connect SCD1 pipeline for SQL Server where I get a mirror of what's live in the source database at the point of ingestion. Now I want to implement a process where I implement a downstream SCD2 table capturing changes for each ingest...
- 693 Views
- 2 replies
- 0 kudos
- 0 kudos
This is exactly the scenario apply_changes_from_snapshot was designed for. It compares consecutive full snapshots and automatically derives inserts, updates, and deletes by absence no delete indicator column needed.
- 0 kudos
- 1827 Views
- 2 replies
- 1 kudos
Fething the catalog and schema which is set in dlt pipeline configuration
I have a dlt pipeline and the notebook which is running on the dlt pipeline has some requirements.I want to get the catalog and schema which is set my dlt pipeline. Reason for it: I have to specify my volume files paths etc and my volume is on the sa...
- 1827 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @ashraf1395 Can you try this to get the catalog and schema set by your DLT pipeline in the notebookcatalog = spark.conf.get("pipelines.catalog")schema = spark.conf.get("pipelines.schema")
- 1 kudos
- 727 Views
- 2 replies
- 1 kudos
Resolved! Auto Loader on UC Volumes stopped resolving wildcards
The following spark.readStream / cloudFiles configuration was confirmed working on2026-04-30, but stopped working on 2026-05-26. No code or config changes were madebetween these dates, so I assume something was changed implicitly on the Databricks si...
- 727 Views
- 2 replies
- 1 kudos
- 1 kudos
We are facing a similar issue, not limited to Autoloader but also affecting DLT pipelines and classic ETL job. The behavior is intermittent, jobs run fine and then fail unexpectedly, though they typically succeed on retry if retries are enabled. We t...
- 1 kudos
- 2276 Views
- 1 replies
- 0 kudos
Resolved! Serverless compute outbound IP whitelisting for external API calls
I’m trying to understand the networking implications of moving some logic to Databricks Serverless / SDP.My current setup is a notebook running as a job on classic compute, and this works because outbound traffic goes through a NAT Gateway, so we can...
- 2276 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @mnissen1337, Your understanding is basically right. With classic compute, the workload runs in your own VNet/VPC, so using your own NAT Gateway to present a stable public egress IP is a standard pattern. With serverless, the compute runs in the D...
- 0 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 584 | |
| 349 | |
| 287 |
