- 41 Views
- 1 replies
- 0 kudos
Cluster environment variables not visible to Java Dataset.foreachPartition / mapPartitions
We run a Spark JAR task (Java) on a standard access mode cluster. Credentials come from cluster environmentvariables set as secret references in the cluster's Environment variables field, e.g.MY_PASSWORD={{secrets/<scope>/<key>}}.The variables are vi...
- 41 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @capoorhimanshu On standard access mode this is expected. Scala/Java closures (foreachPartition, mapPartitions, UDFs) don't run in the executor JVM where the cluster env vars are set. They run in an isolated per-user sandbox process with a fixe...
- 0 kudos
- 130 Views
- 4 replies
- 1 kudos
Serverless notebook: UC volume PUT via SQL connector fails *.storage.azuredatabricks.net resolves
From Azure Databricks serverless notebook compute (East US), uploading a file to a Unity Catalog volume with the Databricks SQL Connector PUT ... INTO '/Volumes/...' fails at TCP connect with [Errno 111] Connection refused. The warehouse accepts the ...
- 130 Views
- 4 replies
- 1 kudos
- 1 kudos
I tested this independently and was able to reproduce a very similar failure pattern on AWS us-east-2, using a Databricks serverless.Here are the results against the same UC Volume:Upload method ResultDirect /Volumes/.. writePASS (byte readback verif...
- 1 kudos
- 38 Views
- 0 replies
- 0 kudos
Avoid AWS SDK 1.x dependencies in cluster classpaths
Hi, As part of an AWS well architected review, we received a recommendation to stop using AWS SDK 1.x since this version has been deprecated since end of 2025. A cloud trail search showed that this lib is still in use in our data AWS accounts. In the...
- 38 Views
- 0 replies
- 0 kudos
- 3529 Views
- 4 replies
- 2 kudos
Spark DataFrame Checkpoint
Good morning,I am having a difficulty when trying to checkpoint a PySpark DataFrame.The DataFrame is not involved in a DLT pipeline so I am using the df.checkpoint(eager=True) command, to truncate the logical plan of df and materialize it as files wi...
- 3529 Views
- 4 replies
- 2 kudos
- 2 kudos
Using a Unity Catalog Volume to write and read back the DataFrame seems like a practical alternative to avoid mounting cloud storage. Just make sure to account for the extra I/O overhead and schema preservation when reloading the data.
- 2 kudos
- 389 Views
- 4 replies
- 3 kudos
Resolved! Anyone migrated from legacy CDF to Auto CDF yet?
Databricks mentions in the DBR 19 release notes that Auto CDF removes write-time overhead and can make MERGE and UPDATE operations about 15% faster on tables queried for changesFor anyone who has moved an existing pipeline from legacy CDF to Auto CDF...
- 389 Views
- 4 replies
- 3 kudos
- 3 kudos
Thanks @ThomazNeto and @Aravind_Reddy for the inputs and suggestions.I did some additional testing around one of the migration concerns raised here. After enabling the Auto CDF prerequisites, I have unset delta.enableChangeDataFeed on the source tabl...
- 3 kudos
- 142 Views
- 3 replies
- 0 kudos
Databricks Unity Catalog Foreign Catalog – Timezone Mismatch with AWS RDS PostgreSQL Affecting Query
Hi Databricks Community,I'm facing a timezone-related issue with a Unity Catalog Foreign Catalog connected to AWS RDS PostgreSQL and would appreciate some guidance on the best way to handle it centrally.Current Setup:Source: AWS RDS PostgreSQL (confi...
- 142 Views
- 3 replies
- 0 kudos
- 0 kudos
@vkondepati I spent some time reproducing a similar scenario using Neon PostgreSQL connected to Databricks through Unity Catalog Lakehouse Federation and found a few observations that might help explain the timezone/date-aggregation mismatch.1. Postg...
- 0 kudos
- 1337 Views
- 3 replies
- 4 kudos
Resolved! Context engineering for Genie
Hello Team,I have a client who is already using Genie One and Genie Agents, and I want to help them build a strong context layer to improve the quality and accuracy of responses.I'm looking for guidance on the overall approach and best practices for ...
- 1337 Views
- 3 replies
- 4 kudos
- 4 kudos
Hi @Niyojit I’m about to start my journey building a context layer. I’m following the documentation Thomaz shared and would really appreciate any learnings or insights you’ve gained since July.
- 4 kudos
- 2571 Views
- 15 replies
- 2 kudos
Unable to Enable Unity Catalog – Azure Managed Identity Credential Not Found
Hi Databricks Community,I am trying to enable Unity Catalog in my Azure Databricks account. As part of the setup, I created an Azure Databricks Access Connector and assigned its managed identity the required permissions on my ADLS Gen2 storage accoun...
- 2571 Views
- 15 replies
- 2 kudos
- 2 kudos
i have created access connector,storage account and databricks resource in same region.Reader and Storage Actions Blob Data Operator access are given.still getting below error Azure Managed Identity Credential with Access Connector Id /subscriptions/...
- 2 kudos
- 305 Views
- 4 replies
- 0 kudos
Feature Request: Cross-job dependencies with business-date awareness
SummaryWorkflows can chain tasks inside one job, but there is no native way to say: “Run job X once per day, after jobs A, B, C, D and E have each completed successfully for the same business date, even though they are independent jobs with their own...
- 305 Views
- 4 replies
- 0 kudos
- 0 kudos
@indradbxlearn Two suggestions for the open items. First, make the control-table write a standard final task in each dimension job, so the job can't succeed without writing its row. Second, for reruns: count repairs that finish before the fact load, ...
- 0 kudos
- 194 Views
- 4 replies
- 4 kudos
Resolved! Materialized View with Incremental Refresh - Change Data Feed required on source tables?
Question:To create a materialized view with incremental refresh, is the Delta Lake feature Change Data Feed required on all of the source tables that the materialized view consumes? Or, is it not required for the creation of the materialized view, bu...
- 194 Views
- 4 replies
- 4 kudos
- 4 kudos
@johnb1 - Hello Johnb1 - CDF is not required, either to create the materialized view or to refresh it incrementally. The current Databricks docs (updated October 2026) list it only as a recommendation for better performance.Row tracking is what can ...
- 4 kudos
- 940 Views
- 6 replies
- 7 kudos
Resolved! Unity Catalog v.2.0?
Are there plans for a v.2.0 of Unity Catalog? I find the organization of tables in databricks to be very constrained and inflexible. Given that UC tables have their origins in data lakes, you would think they would have brought a lot more flexibile...
- 940 Views
- 6 replies
- 7 kudos
- 7 kudos
Fair points, especially for data coming from systems with richer naming. I can't speak to the roadmap, but here's what works for us today:Aliases: Create views in a consumer-facing catalog, like reporting.sales.outside_sales, that point to the real t...
- 7 kudos
- 2434 Views
- 2 replies
- 4 kudos
How to Unit Test Spark Declarative Pipelines
Every engineer who ships application code writes unit tests for it. Yet the data pipeline that feeds the same application is often left with a manual run and a dashboard that someone eventually notices is wrong. The gap has been more about tooling th...
- 2434 Views
- 2 replies
- 4 kudos
- 4 kudos
Great article. The native TestPipeline and test_spark APIs are a promising step towards first-class unit testing for SDP.One potential enhancement would be execution outside the Lakeflow Pipelines Editor.As documented, native test execution is curren...
- 4 kudos
- 191 Views
- 5 replies
- 1 kudos
Salesforce - "Unexpected error occurred" while connecting with Databricks
Hi everyone,I am using Salesforce trail version. I am getting error which shows "Unexpected error occurred" while connecting it with Databricks free edition, However, I already passed the necessary OAuth with the credentials.Please advise what action...
- 191 Views
- 5 replies
- 1 kudos
- 1 kudos
@J-DataCould you as-well post the Error details to see if its connection issue?
- 1 kudos
- 481 Views
- 5 replies
- 3 kudos
Feature request: let DABs alert resources load query_text from an external .sql file
Hi everyone,We manage SQL alerts with Declarative Automation Bundles and like having them in version control. One thing we'd like to improve: the alert's SQL query has to be written inline in the YAML resource definition (query_text).The problemOur a...
- 481 Views
- 5 replies
- 3 kudos
- 3 kudos
Short answer: You can get "SQL in a .sql file" today, without waiting for a query_file field, by defining the alerts with Python for Declarative Automation Bundles [1]. A small load_resources() function reads each .sql file and passes its contents as...
- 3 kudos
- 11136 Views
- 3 replies
- 4 kudos
Why database/table names are in lower case ?
Hello,When I run this code :CREATE DATABASE BackOfficeI see the database like this :backofficeWhy everything is in lower case ?Is it possible to configure Databricks in order to keep the real name ?Thanks.
- 11136 Views
- 3 replies
- 4 kudos
- 4 kudos
It is managed by hive metastore as you can put it in different databases is saver this way as some database are Case Sensitive and some not (you can easily test it with standard WHERE syntax).Probably you could change it with some hive settings but i...
- 4 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 586 | |
| 349 | |
| 287 |
