cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

vkondepati
by • Databricks Partner
  • 62 Views
  • 1 replies
  • 0 kudos

Databricks Unity Catalog Foreign Catalog – Timezone Mismatch with AWS RDS PostgreSQL Affecting Query

Hi Databricks Community,I'm facing a timezone-related issue with a Unity Catalog Foreign Catalog connected to AWS RDS PostgreSQL and would appreciate some guidance on the best way to handle it centrally.Current Setup:Source: AWS RDS PostgreSQL (confi...

  • 62 Views
  • 1 replies
  • 0 kudos
Latest Reply
DoTA
Valued Contributor III
  • 0 kudos

Short version: I could not find a documented timezone setting at the foreign catalog, connection or warehouse level, so I would plan on fixing it once in a governed layer rather than per query. Treat the details below as things to verify on your side...

  • 0 kudos
Shubh5073
by • New Contributor
  • 2526 Views
  • 15 replies
  • 2 kudos

Unable to Enable Unity Catalog – Azure Managed Identity Credential Not Found

Hi Databricks Community,I am trying to enable Unity Catalog in my Azure Databricks account. As part of the setup, I created an Azure Databricks Access Connector and assigned its managed identity the required permissions on my ADLS Gen2 storage accoun...

  • 2526 Views
  • 15 replies
  • 2 kudos
Latest Reply
deshpanderutuja
New Contributor II
  • 2 kudos

i have created access connector,storage account and databricks resource in same region.Reader and Storage Actions Blob Data Operator access are given.still getting below error Azure Managed Identity Credential with Access Connector Id /subscriptions/...

  • 2 kudos
14 More Replies
indradbxlearn
by • New Contributor II
  • 275 Views
  • 4 replies
  • 0 kudos

Feature Request: Cross-job dependencies with business-date awareness

SummaryWorkflows can chain tasks inside one job, but there is no native way to say: “Run job X once per day, after jobs A, B, C, D and E have each completed successfully for the same business date, even though they are independent jobs with their own...

  • 275 Views
  • 4 replies
  • 0 kudos
Latest Reply
SumeshKashyap
Valued Contributor II
  • 0 kudos

@indradbxlearn Two suggestions for the open items. First, make the control-table write a standard final task in each dimension job, so the job can't succeed without writing its row. Second, for reruns: count repairs that finish before the fact load, ...

  • 0 kudos
3 More Replies
johnb1
by • Contributor
  • 148 Views
  • 4 replies
  • 0 kudos

Materialized View with Incremental Refresh - Change Data Feed required on source tables?

Question:To create a materialized view with incremental refresh, is the Delta Lake feature Change Data Feed required on all of the source tables that the materialized view consumes? Or, is it not required for the creation of the materialized view, bu...

  • 148 Views
  • 4 replies
  • 0 kudos
Latest Reply
indradbxlearn
New Contributor II
  • 0 kudos

@johnb1 - Hello Johnb1 -  CDF is not required, either to create the materialized view or to refresh it incrementally. The current Databricks docs (updated October 2026) list it only as a recommendation for better performance.Row tracking is what can ...

  • 0 kudos
3 More Replies
DB1To3
by • Valued Contributor
  • 899 Views
  • 6 replies
  • 7 kudos

Resolved! Unity Catalog v.2.0?

Are there plans for a v.2.0 of Unity Catalog?  I find the organization of tables in databricks to be very constrained and inflexible.  Given that UC tables have their origins in data lakes, you would think they would have brought a lot more flexibile...

  • 899 Views
  • 6 replies
  • 7 kudos
Latest Reply
LiresaFerizaj
Contributor
  • 7 kudos

Fair points, especially for data coming from systems with richer naming. I can't speak to the roadmap, but here's what works for us today:Aliases: Create views in a consumer-facing catalog, like reporting.sales.outside_sales, that point to the real t...

  • 7 kudos
5 More Replies
LingeshK
by • Databricks Employee
  • 2363 Views
  • 2 replies
  • 4 kudos

How to Unit Test Spark Declarative Pipelines

Every engineer who ships application code writes unit tests for it. Yet the data pipeline that feeds the same application is often left with a manual run and a dashboard that someone eventually notices is wrong. The gap has been more about tooling th...

00-etl-diagram-excalidraw.png 03-v2-pipeline-dag-closeup.png 13-v2-gold-table-wrong-numbers.png 12-v2-run-pipeline-complete-green.png
  • 2363 Views
  • 2 replies
  • 4 kudos
Latest Reply
data_pulse
Contributor
  • 4 kudos

Great article. The native TestPipeline and test_spark APIs are a promising step towards first-class unit testing for SDP.One potential enhancement would be execution outside the Lakeflow Pipelines Editor.As documented, native test execution is curren...

  • 4 kudos
1 More Replies
Aamir2028
by • New Contributor
  • 170 Views
  • 5 replies
  • 1 kudos

Salesforce - "Unexpected error occurred" while connecting with Databricks

Hi everyone,I am using Salesforce trail version. I am getting error which shows "Unexpected error occurred" while connecting it with Databricks free edition, However, I already passed the necessary OAuth with the credentials.Please advise what action...

Aamir2028_0-1792767742215.png
  • 170 Views
  • 5 replies
  • 1 kudos
Latest Reply
data_pulse
Contributor
  • 1 kudos

 @J-DataCould you as-well post the Error details to see if its connection issue?

  • 1 kudos
4 More Replies
Pavel_Sinkevich
by • New Contributor III
  • 440 Views
  • 5 replies
  • 2 kudos

Feature request: let DABs alert resources load query_text from an external .sql file

Hi everyone,We manage SQL alerts with Declarative Automation Bundles and like having them in version control. One thing we'd like to improve: the alert's SQL query has to be written inline in the YAML resource definition (query_text).The problemOur a...

  • 440 Views
  • 5 replies
  • 2 kudos
Latest Reply
VladTamas
New Contributor II
  • 2 kudos

Short answer: You can get "SQL in a .sql file" today, without waiting for a query_file field, by defining the alerts with Python for Declarative Automation Bundles [1]. A small load_resources() function reads each .sql file and passes its contents as...

  • 2 kudos
4 More Replies
wyzer
by • Contributor III
  • 11123 Views
  • 3 replies
  • 4 kudos

Why database/table names are in lower case ?

Hello,When I run this code :CREATE DATABASE BackOfficeI see the database like this :backofficeWhy everything is in lower case ?Is it possible to configure Databricks in order to keep the real name ?Thanks.

  • 11123 Views
  • 3 replies
  • 4 kudos
Latest Reply
Hubert-Dudek
Databricks MVP
  • 4 kudos

It is managed by hive metastore as you can put it in different databases is saver this way as some database are Case Sensitive and some not (you can easily test it with standard WHERE syntax).Probably you could change it with some hive settings but i...

  • 4 kudos
2 More Replies
dbernstein_tp
by • Contributor II
  • 361 Views
  • 12 replies
  • 3 kudos

MS Fabric Direct Lake, works with delta tables only?

Lately we started using Direct Lake to use the results of our ingestion pipelines (gold tables) in PBI dashboards. This is working ok but has one annoying and troublesome issue. Which is: the gold tables are the end result of declarative pipelines so...

  • 361 Views
  • 12 replies
  • 3 kudos
Latest Reply
Jen_Evangelista
New Contributor II
  • 3 kudos

There's no preview toggle for this, which is why you don't see it in the workspace or account previews. For an MV, Compatibility Mode is set with the two table properties Louis showed, when the MV is created. The docs only document ALTER TABLE for ma...

  • 3 kudos
11 More Replies
Aju_Antony
by • New Contributor
  • 157 Views
  • 2 replies
  • 0 kudos

Clarification on use_describe_as_json_for_relation_metadata and system.information_schema dependency

Hi Databricks Support Team,We are evaluating whether dbt can execute successfully without access to system.information_schema in our environment, due to metadata access restrictions.Our current setup uses:dbt-core: 1.12.3dbt-databricks: 1.12.5Configu...

  • 157 Views
  • 2 replies
  • 0 kudos
Latest Reply
SumeshKashyap
Valued Contributor II
  • 0 kudos

Hi @Aju_Antony, were you able to try the get_uc_tables_sql override in your POC? Quick way to confirm it works: run dbt run with --debug and check that no query in the log references system.information_schema.If that unblocked you, please click Accep...

  • 0 kudos
1 More Replies
ankit2604
by • New Contributor II
  • 838 Views
  • 14 replies
  • 12 kudos

Datavault implemention on Databricks

Hi everyone,I'm building a Data Vault 2.0 architecture entirely natively in Databricks from scratch. We have specifically decided not to use dbt.While Raw Vault ingestion is straightforward with Spark/Delta, I'm looking for architectural advice on co...

  • 838 Views
  • 14 replies
  • 12 kudos
Latest Reply
Louis_Frolio
Databricks Employee
  • 12 kudos

Hello @ankit2604, I took a look at both internal and external documentation and here is what I found. Thanks @balajij8, @Satyasai, @Jen_Evangelista and @data_pulse for doing most of the work here. The shape they've described is right: streaming table...

  • 12 kudos
13 More Replies
harsh120
by • New Contributor III
  • 344 Views
  • 6 replies
  • 5 kudos

Table does not support batch scan.

After moving our incremental load jobs from DBR 16.4 (Spark 3.5) to DBR 18 (Spark 4.1.0), they fail with this error: [UNSUPPORTED_FEATURE.TABLE_OPERATION] The feature is not supported: Table `app_dev`.`vw_period` does not support batch scan. Please c...

  • 344 Views
  • 6 replies
  • 5 kudos
Latest Reply
Louis_Frolio
Databricks Employee
  • 5 kudos

Hello @harsh120, I took a look at both internal and external documentation and here is what I found. Building on @data_pulse and @Jen_Evangelista: RemoteSparkConnectScan is the key clue, and this doesn't look like MAP(), isEmpty(), or ANSI mode being...

  • 5 kudos
5 More Replies
cdn_yyz_yul
by • Valued Contributor
  • 1595 Views
  • 18 replies
  • 4 kudos

Change data feed from a materialized view

Hi everyone,Trying the CDF on materialized view (declarative pipeline) feature that is currently in Beta.- Configuration:** Verified the MVs (materialized veiws) have: 'delta.enableRowTracking'SHOW TBLPROPERTIES my_mv ('delta.enableRowTracking');  re...

  • 1595 Views
  • 18 replies
  • 4 kudos
Latest Reply
Louis_Frolio
Databricks Employee
  • 4 kudos

Hello @cdn_yyz_yul, Thanksto @Jen_Evangelista for the pipeline-config versus table-property framing and to @data_pulse for the Free Edition repro. On the property question: I can't point to a doc line, but the evidence says it's expected that pipelin...

  • 4 kudos
17 More Replies
shubham_1507
by • New Contributor
  • 209 Views
  • 3 replies
  • 2 kudos

Is Genie Ontology Available in Databricks Free Edition?

Hi Community,I wanted to confirm whether Genie Ontology is available in the Databricks Free Edition or if it is exclusive to paid editions.If it is supported in the Free Edition, could someone please guide me on how to access and enable this feature ...

  • 209 Views
  • 3 replies
  • 2 kudos
Latest Reply
data_pulse
Contributor
  • 2 kudos

@shubham_1507 Can validate this directly in Databricks Free Edition workspace.Open Genie One and ask:“Is Genie Ontology available in this workspace?” I have tested in Free edition workspace as below, In the expanded Thinking steps, Genie invoked sear...

  • 2 kudos
2 More Replies
Labels