cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Kayla
by Valued Contributor II
  • 150 Views
  • 3 replies
  • 1 kudos

Disable spark transformations inside try except warning

I'm wondering if there's a way to suppress certain errors? I have a lot of spammy yellow underlining from this one warning, and it doesn't seem that the warning behaves properly. Transformation inside try/except block is lazy, fine, I have actions on...

Kayla_0-1785436282119.png
  • 150 Views
  • 3 replies
  • 1 kudos
Latest Reply
masonreed11
New Contributor II
  • 1 kudos

If your action (count(), collect(), show(), etc.) is already inside the try block, this sounds like a false positive. The warning is likely from a Databricks inspection rather than the Flint time-series library. Check if inspection SCPAP005 can be di...

  • 1 kudos
2 More Replies
KKo
by Contributor III
  • 96 Views
  • 1 replies
  • 0 kudos

Real time pipelines

How do you design high-volume real-time pipelines that need to stay reliable under load spikes?

  • 96 Views
  • 1 replies
  • 0 kudos
Latest Reply
ThomazNeto
Databricks Partner
  • 0 kudos

Big question, but in my experience it comes down to five decisions. Opinionated take:1. Put a buffer in front, always. Kafka/Kinesis/Event Hubs between producers and Spark. The broker absorbs the spike; your pipeline drains at its own pace. A pipelin...

  • 0 kudos
binlogreader
by New Contributor II
  • 123 Views
  • 2 replies
  • 1 kudos

Streaming tables vs materialized views in declarative pipelines: what actually decides the choice?

I am trying to settle a design default for our declarative pipelines, and I would like to hear how others decide between streaming tables and materialized views when a table could be built either way.When I started with streaming tables, I thought th...

  • 123 Views
  • 2 replies
  • 1 kudos
Latest Reply
ThomazNeto
Databricks Partner
  • 1 kudos

Good points, Brahmareddy — and I think we mostly agree, with one nuance worth teasing out on the cross-team part.A published streaming table already *is* a stable Delta table from the reader's perspective — consumers query it by its UC name and never...

  • 1 kudos
1 More Replies
ali_lmalki
by New Contributor
  • 160 Views
  • 2 replies
  • 1 kudos

Genie Ontology

One question that came to mind regarding Genie Ontology is its level of automated curation. From what I've seen, Databricks appears to be taking a different approach from many vendors, where ontologies are often manually designed.Are there meaningful...

  • 160 Views
  • 2 replies
  • 1 kudos
Latest Reply
Lu_Wang_ENB_DBX
Databricks Employee
  • 1 kudos

Answers: Automated curation trade-off: the upside is low setup; the downside is less explicit control. Genie ranks and conflict-resolves snippets automatically, so if your sources are inconsistent or ambiguous, you still need to review and clean up ...

  • 1 kudos
1 More Replies
lausikh
by New Contributor
  • 130 Views
  • 2 replies
  • 0 kudos

What happens to with SCD2 tables if the source pipeline performs a Full Refresh?

Hi everyone,I'm looking for clarification on how Lakeflow/DLT behaves in the following scenario.We have two independent pipelines:Pipeline 1 (Ingestion)Ingests Salesforce objects into Delta tables.These tables represent the current snapshot of Salesf...

  • 130 Views
  • 2 replies
  • 0 kudos
Latest Reply
AbhilashNagilla
Databricks Employee
  • 0 kudos

The pattern is documented and you are right not to full refresh pipeline 2, but as written the deletion detection can never fire, and the full refresh is not why. spark.readStream.table("raw.contact") makes contact append-only, so no key ever leaves ...

  • 0 kudos
1 More Replies
yutmurakami
by New Contributor
  • 184 Views
  • 3 replies
  • 1 kudos

Databricks Apps - "App Not Available" error with locationId parameter missing

Hello,I'm experiencing an issue with Databricks Apps where I cannot access my deployed app from a browser, even though the app appears to be running successfully.**Symptoms:**- Browser shows "App Not Available" error when accessing the app URL- Conso...

  • 184 Views
  • 3 replies
  • 1 kudos
Latest Reply
ThomazNeto
Databricks Partner
  • 1 kudos

Nice progress — with compute ACTIVE and the app RUNNING, we've eliminated the original cause and narrowed this down to the auth/access layer. A few thoughts:First, one thing in your test that's easy to over-read: the Python requests behavior is actua...

  • 1 kudos
2 More Replies
Niyojit
by New Contributor II
  • 171 Views
  • 2 replies
  • 2 kudos

Resolved! Context engineering for Genie

Hello Team,I have a client who is already using Genie One and Genie Agents, and I want to help them build a strong context layer to improve the quality and accuracy of responses.I'm looking for guidance on the overall approach and best practices for ...

  • 171 Views
  • 2 replies
  • 2 kudos
Latest Reply
Niyojit
New Contributor II
  • 2 kudos

Thanks!!

  • 2 kudos
1 More Replies
ChristianRRL
by Honored Contributor II
  • 216 Views
  • 3 replies
  • 2 kudos

CREATE CONNECTION - Support for Community Connections?

Hi there, as the title asks, I'm wondering how Databricks Community Connections are being supported. For example, when I try to make a fresh connection via Spark SQL commands, I get an error because `COMMUNITY` type is not supported.However, when I c...

ChristianRRL_0-1785276761138.png ChristianRRL_1-1785276787254.png ChristianRRL_4-1785277045154.png ChristianRRL_3-1785276855186.png
  • 216 Views
  • 3 replies
  • 2 kudos
Latest Reply
adnan_alvee
Databricks Employee
  • 2 kudos

Hi @ChristianRRL  I tried with FHIR and it worked. The process is the same for every source. To do GitHub instead of the FHIR example, you change exactly three things, the connector name (github), the credential (a GitHub token), and the list of tabl...

  • 2 kudos
2 More Replies
skipowder99
by Databricks Partner
  • 121 Views
  • 2 replies
  • 0 kudos

Collation Change Planning & Fixing

We recently decided to change from the default collation of UTF8_BINARY to UTF8_LCASE and there are many issues.  We attempted to cascade the collation defaults to all the Catalogs, Schemas, Tables, Views and Columns, but we still seem to have expres...

  • 121 Views
  • 2 replies
  • 0 kudos
Latest Reply
balajij8
Esteemed Contributor
  • 0 kudos

Hi skipowder99,When you set a DEFAULT COLLATION at the catalog or schema level, it is forward looking. It does not retroactively apply to existing tables, columns or views. It leaves you with a split environment the newly created objects default to U...

  • 0 kudos
1 More Replies
vkondepati
by Databricks Partner
  • 90 Views
  • 0 replies
  • 0 kudos

V.E.N.K.A.T Framework™

Hello Databricks Community,I recently published the V.E.N.K.A.T Framework™, an enterprise architecture model for the Agentic AI era.The idea is simple:As AI moves from generating insights to taking actions, the architecture underneath becomes critica...

  • 90 Views
  • 0 replies
  • 0 kudos
GJ2
by New Contributor II
  • 28615 Views
  • 16 replies
  • 2 kudos

Install the ODBC Driver 17 for SQL Server

Hi,I am not a Data Engineer, I want to connect to ssas. It looks like it can be connected through pyodbc. however looks like  I need to install "ODBC Driver 17 for SQL Server" using the following command. How do i install the driver on the cluster an...

GJ2_1-1739798450883.png
  • 28615 Views
  • 16 replies
  • 2 kudos
Latest Reply
lilydays
New Contributor
  • 2 kudos

If you're only connecting to SSAS, installing the ODBC Driver 17 for SQL Server is usually the first step, but the exact method depends on your cluster environment and operating system. If it's a managed cluster, you may need admin access or an scrip...

  • 2 kudos
15 More Replies
batch_bender
by New Contributor II
  • 152 Views
  • 2 replies
  • 1 kudos

Folder structure for Autoloader and Declarative Pipelines

We have six years of Parquet files organised under yyyy/mm/dd/, with around 100 table files in each daily folder. We want to process these 100 tables using Auto Loader in a Declarative Pipeline with the AUTO CDC API.If each table reads the same root ...

  • 152 Views
  • 2 replies
  • 1 kudos
Latest Reply
GabFernandes
New Contributor III
  • 1 kudos

Hi @batch_bender ,Yes, pointing Auto Loader to the root path with a filename wildcard for every table will cause significant file discovery overhead, as each stream independently crawls the entire directory tree (yyyy/mm/dd/).Best Practice Recommenda...

  • 1 kudos
1 More Replies
matko
by New Contributor
  • 111 Views
  • 1 replies
  • 0 kudos

oreign Iceberg tables via HMS federation fail with browse_only_table on all compute — request Public

Per your documentation (hms-federation-concepts), Apache Iceberg tables in federated external Hive metastores are Public Preview on Databricks Runtime 16.2+, with no enablement steps listed. On our workspace, every read of a federated Iceberg table f...

  • 111 Views
  • 1 replies
  • 0 kudos
Latest Reply
balajij8
Esteemed Contributor
  • 0 kudos

Hi matko,The underlying issue here is mostly a missing backend feature enablement for the workspace. Databricks frequently gates these newer federation capabilities behind workspace-level or account-level enablement during the preview phase.Unity Cat...

  • 0 kudos
Sam500
by New Contributor III
  • 129 Views
  • 1 replies
  • 0 kudos

Resolved! Multi-Engine Lakehouses

Hi experts,I recently came across an interesting discussion titled “Are enterprises moving from ‘Data Lakehouse’ to ‘Agentic Lakehouse’?” https://community.databricks.com/t5/data-engineering/are-enterprises-moving-from-quot-data-lakehouse-quot-to-quo...

  • 129 Views
  • 1 replies
  • 0 kudos
Latest Reply
balajij8
Esteemed Contributor
  • 0 kudos

Hi Sam500,The multi engine pattern usually comes to establishing interoperability at the storage format level while maintaining a single, unified governance layer. You handle this primarily through Unity Catalog (UC) and format flexibility such as De...

  • 0 kudos
Labels