cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

lausikh
by New Contributor
  • 40 Views
  • 1 replies
  • 0 kudos

What happens to with SCD2 tables if the source pipeline performs a Full Refresh?

Hi everyone,I'm looking for clarification on how Lakeflow/DLT behaves in the following scenario.We have two independent pipelines:Pipeline 1 (Ingestion)Ingests Salesforce objects into Delta tables.These tables represent the current snapshot of Salesf...

  • 40 Views
  • 1 replies
  • 0 kudos
Latest Reply
ThomazNeto
New Contributor II
  • 0 kudos

The concept is supported — but there's one detail in your code that's likely the real risk, and it's not the full refresh itself.1. Deletion detection: yes, that's literally what the API is designed for. AUTO CDC FROM SNAPSHOT (the new name for apply...

  • 0 kudos
yutmurakami
by Visitor
  • 110 Views
  • 3 replies
  • 1 kudos

Databricks Apps - "App Not Available" error with locationId parameter missing

Hello,I'm experiencing an issue with Databricks Apps where I cannot access my deployed app from a browser, even though the app appears to be running successfully.**Symptoms:**- Browser shows "App Not Available" error when accessing the app URL- Conso...

  • 110 Views
  • 3 replies
  • 1 kudos
Latest Reply
ThomazNeto
New Contributor II
  • 1 kudos

Nice progress — with compute ACTIVE and the app RUNNING, we've eliminated the original cause and narrowed this down to the auth/access layer. A few thoughts:First, one thing in your test that's easy to over-read: the Python requests behavior is actua...

  • 1 kudos
2 More Replies
Niyojit
by New Contributor II
  • 131 Views
  • 2 replies
  • 2 kudos

Resolved! Context engineering for Genie

Hello Team,I have a client who is already using Genie One and Genie Agents, and I want to help them build a strong context layer to improve the quality and accuracy of responses.I'm looking for guidance on the overall approach and best practices for ...

  • 131 Views
  • 2 replies
  • 2 kudos
Latest Reply
Niyojit
New Contributor II
  • 2 kudos

Thanks!!

  • 2 kudos
1 More Replies
ali_lmalki
by Visitor
  • 91 Views
  • 1 replies
  • 0 kudos

Genie Ontology

One question that came to mind regarding Genie Ontology is its level of automated curation. From what I've seen, Databricks appears to be taking a different approach from many vendors, where ontologies are often manually designed.Are there meaningful...

  • 91 Views
  • 1 replies
  • 0 kudos
Latest Reply
ThomazNeto
New Contributor II
  • 0 kudos

Been digging into this since DAIS, so I'll share what I've found — with the big caveat up front that the Ontology is still in preview and the public docs are thin. Some of your questions genuinely don't have documented answers yet, and I think that's...

  • 0 kudos
Kayla
by Valued Contributor II
  • 80 Views
  • 1 replies
  • 0 kudos

Disable spark transformations inside try except warning

I'm wondering if there's a way to suppress certain errors? I have a lot of spammy yellow underlining from this one warning, and it doesn't seem that the warning behaves properly. Transformation inside try/except block is lazy, fine, I have actions on...

Kayla_0-1785436282119.png
  • 80 Views
  • 1 replies
  • 0 kudos
Latest Reply
adnan_alvee
Databricks Employee
  • 0 kudos

Can u try this in your toml file? [tool.databricks.flint] ignore = ["SCPAP005"]

  • 0 kudos
ChristianRRL
by Honored Contributor II
  • 185 Views
  • 3 replies
  • 2 kudos

CREATE CONNECTION - Support for Community Connections?

Hi there, as the title asks, I'm wondering how Databricks Community Connections are being supported. For example, when I try to make a fresh connection via Spark SQL commands, I get an error because `COMMUNITY` type is not supported.However, when I c...

ChristianRRL_0-1785276761138.png ChristianRRL_1-1785276787254.png ChristianRRL_4-1785277045154.png ChristianRRL_3-1785276855186.png
  • 185 Views
  • 3 replies
  • 2 kudos
Latest Reply
adnan_alvee
Databricks Employee
  • 2 kudos

Hi @ChristianRRL  I tried with FHIR and it worked. The process is the same for every source. To do GitHub instead of the FHIR example, you change exactly three things, the connector name (github), the credential (a GitHub token), and the list of tabl...

  • 2 kudos
2 More Replies
skipowder99
by Databricks Partner
  • 95 Views
  • 2 replies
  • 0 kudos

Collation Change Planning & Fixing

We recently decided to change from the default collation of UTF8_BINARY to UTF8_LCASE and there are many issues.  We attempted to cascade the collation defaults to all the Catalogs, Schemas, Tables, Views and Columns, but we still seem to have expres...

  • 95 Views
  • 2 replies
  • 0 kudos
Latest Reply
balajij8
Esteemed Contributor
  • 0 kudos

Hi skipowder99,When you set a DEFAULT COLLATION at the catalog or schema level, it is forward looking. It does not retroactively apply to existing tables, columns or views. It leaves you with a split environment the newly created objects default to U...

  • 0 kudos
1 More Replies
vkondepati
by Databricks Partner
  • 69 Views
  • 0 replies
  • 0 kudos

V.E.N.K.A.T Framework™

Hello Databricks Community,I recently published the V.E.N.K.A.T Framework™, an enterprise architecture model for the Agentic AI era.The idea is simple:As AI moves from generating insights to taking actions, the architecture underneath becomes critica...

  • 69 Views
  • 0 replies
  • 0 kudos
GJ2
by New Contributor II
  • 28581 Views
  • 16 replies
  • 2 kudos

Install the ODBC Driver 17 for SQL Server

Hi,I am not a Data Engineer, I want to connect to ssas. It looks like it can be connected through pyodbc. however looks like  I need to install "ODBC Driver 17 for SQL Server" using the following command. How do i install the driver on the cluster an...

GJ2_1-1739798450883.png
  • 28581 Views
  • 16 replies
  • 2 kudos
Latest Reply
lilydays
Visitor
  • 2 kudos

If you're only connecting to SSAS, installing the ODBC Driver 17 for SQL Server is usually the first step, but the exact method depends on your cluster environment and operating system. If it's a managed cluster, you may need admin access or an scrip...

  • 2 kudos
15 More Replies
batch_bender
by New Contributor II
  • 114 Views
  • 2 replies
  • 1 kudos

Folder structure for Autoloader and Declarative Pipelines

We have six years of Parquet files organised under yyyy/mm/dd/, with around 100 table files in each daily folder. We want to process these 100 tables using Auto Loader in a Declarative Pipeline with the AUTO CDC API.If each table reads the same root ...

  • 114 Views
  • 2 replies
  • 1 kudos
Latest Reply
GabFernandes
New Contributor III
  • 1 kudos

Hi @batch_bender ,Yes, pointing Auto Loader to the root path with a filename wildcard for every table will cause significant file discovery overhead, as each stream independently crawls the entire directory tree (yyyy/mm/dd/).Best Practice Recommenda...

  • 1 kudos
1 More Replies
matko
by New Contributor
  • 78 Views
  • 1 replies
  • 0 kudos

oreign Iceberg tables via HMS federation fail with browse_only_table on all compute — request Public

Per your documentation (hms-federation-concepts), Apache Iceberg tables in federated external Hive metastores are Public Preview on Databricks Runtime 16.2+, with no enablement steps listed. On our workspace, every read of a federated Iceberg table f...

  • 78 Views
  • 1 replies
  • 0 kudos
Latest Reply
balajij8
Esteemed Contributor
  • 0 kudos

Hi matko,The underlying issue here is mostly a missing backend feature enablement for the workspace. Databricks frequently gates these newer federation capabilities behind workspace-level or account-level enablement during the preview phase.Unity Cat...

  • 0 kudos
Sam500
by New Contributor III
  • 85 Views
  • 1 replies
  • 0 kudos

Resolved! Multi-Engine Lakehouses

Hi experts,I recently came across an interesting discussion titled “Are enterprises moving from ‘Data Lakehouse’ to ‘Agentic Lakehouse’?” https://community.databricks.com/t5/data-engineering/are-enterprises-moving-from-quot-data-lakehouse-quot-to-quo...

  • 85 Views
  • 1 replies
  • 0 kudos
Latest Reply
balajij8
Esteemed Contributor
  • 0 kudos

Hi Sam500,The multi engine pattern usually comes to establishing interoperability at the storage format level while maintaining a single, unified governance layer. You handle this primarily through Unity Catalog (UC) and format flexibility such as De...

  • 0 kudos
binlogreader
by New Contributor II
  • 89 Views
  • 2 replies
  • 1 kudos

DLT driver GC pressure during a large initial hydration. Is a bigger driver the only lever?

Last week one of our DLT pipelines spent five hours in RUNNING with nothing landing, and I'd like to compare notes on how people handle the underlying problem, because we found one fix and several dead ends.Setup: A continuous DLT pipeline on classic...

  • 89 Views
  • 2 replies
  • 1 kudos
Latest Reply
masonreed11
New Contributor II
  • 1 kudos

I ran into something similar, and the event log was the only place that made the root cause obvious. I don't believe there's an officially supported setting to limit concurrent DLT flows, so I'd avoid undocumented configs. For large initial backfills...

  • 1 kudos
1 More Replies
Labels