cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Niyojit
by New Contributor II
  • 117 Views
  • 2 replies
  • 2 kudos

Resolved! Context engineering for Genie

Hello Team,I have a client who is already using Genie One and Genie Agents, and I want to help them build a strong context layer to improve the quality and accuracy of responses.I'm looking for guidance on the overall approach and best practices for ...

  • 117 Views
  • 2 replies
  • 2 kudos
Latest Reply
Niyojit
New Contributor II
  • 2 kudos

Thanks!!

  • 2 kudos
1 More Replies
yutmurakami
by Visitor
  • 80 Views
  • 1 replies
  • 0 kudos

Databricks Apps - "App Not Available" error with locationId parameter missing

Hello,I'm experiencing an issue with Databricks Apps where I cannot access my deployed app from a browser, even though the app appears to be running successfully.**Symptoms:**- Browser shows "App Not Available" error when accessing the app URL- Conso...

  • 80 Views
  • 1 replies
  • 0 kudos
Latest Reply
ThomazNeto
New Contributor II
  • 0 kudos

Hey — I think the answer is hiding in your own checklist, in that last item you marked with a ✓:"App compute was stopped due to workspace or account status"That's not a side observation, that's almost certainly your root cause. In Databricks Apps, *d...

  • 0 kudos
ali_lmalki
by Visitor
  • 82 Views
  • 1 replies
  • 0 kudos

Genie Ontology

One question that came to mind regarding Genie Ontology is its level of automated curation. From what I've seen, Databricks appears to be taking a different approach from many vendors, where ontologies are often manually designed.Are there meaningful...

  • 82 Views
  • 1 replies
  • 0 kudos
Latest Reply
ThomazNeto
New Contributor II
  • 0 kudos

Been digging into this since DAIS, so I'll share what I've found — with the big caveat up front that the Ontology is still in preview and the public docs are thin. Some of your questions genuinely don't have documented answers yet, and I think that's...

  • 0 kudos
Kayla
by Valued Contributor II
  • 76 Views
  • 1 replies
  • 0 kudos

Disable spark transformations inside try except warning

I'm wondering if there's a way to suppress certain errors? I have a lot of spammy yellow underlining from this one warning, and it doesn't seem that the warning behaves properly. Transformation inside try/except block is lazy, fine, I have actions on...

Kayla_0-1785436282119.png
  • 76 Views
  • 1 replies
  • 0 kudos
Latest Reply
adnan_alvee
Databricks Employee
  • 0 kudos

Can u try this in your toml file? [tool.databricks.flint] ignore = ["SCPAP005"]

  • 0 kudos
ChristianRRL
by Honored Contributor II
  • 172 Views
  • 3 replies
  • 2 kudos

CREATE CONNECTION - Support for Community Connections?

Hi there, as the title asks, I'm wondering how Databricks Community Connections are being supported. For example, when I try to make a fresh connection via Spark SQL commands, I get an error because `COMMUNITY` type is not supported.However, when I c...

ChristianRRL_0-1785276761138.png ChristianRRL_1-1785276787254.png ChristianRRL_4-1785277045154.png ChristianRRL_3-1785276855186.png
  • 172 Views
  • 3 replies
  • 2 kudos
Latest Reply
adnan_alvee
Databricks Employee
  • 2 kudos

Hi @ChristianRRL  I tried with FHIR and it worked. The process is the same for every source. To do GitHub instead of the FHIR example, you change exactly three things, the connector name (github), the credential (a GitHub token), and the list of tabl...

  • 2 kudos
2 More Replies
skipowder99
by Databricks Partner
  • 93 Views
  • 2 replies
  • 0 kudos

Collation Change Planning & Fixing

We recently decided to change from the default collation of UTF8_BINARY to UTF8_LCASE and there are many issues.  We attempted to cascade the collation defaults to all the Catalogs, Schemas, Tables, Views and Columns, but we still seem to have expres...

  • 93 Views
  • 2 replies
  • 0 kudos
Latest Reply
balajij8
Esteemed Contributor
  • 0 kudos

Hi skipowder99,When you set a DEFAULT COLLATION at the catalog or schema level, it is forward looking. It does not retroactively apply to existing tables, columns or views. It leaves you with a split environment the newly created objects default to U...

  • 0 kudos
1 More Replies
vkondepati
by Databricks Partner
  • 67 Views
  • 0 replies
  • 0 kudos

V.E.N.K.A.T Framework™

Hello Databricks Community,I recently published the V.E.N.K.A.T Framework™, an enterprise architecture model for the Agentic AI era.The idea is simple:As AI moves from generating insights to taking actions, the architecture underneath becomes critica...

  • 67 Views
  • 0 replies
  • 0 kudos
GJ2
by New Contributor II
  • 28574 Views
  • 16 replies
  • 2 kudos

Install the ODBC Driver 17 for SQL Server

Hi,I am not a Data Engineer, I want to connect to ssas. It looks like it can be connected through pyodbc. however looks like  I need to install "ODBC Driver 17 for SQL Server" using the following command. How do i install the driver on the cluster an...

GJ2_1-1739798450883.png
  • 28574 Views
  • 16 replies
  • 2 kudos
Latest Reply
lilydays
Visitor
  • 2 kudos

If you're only connecting to SSAS, installing the ODBC Driver 17 for SQL Server is usually the first step, but the exact method depends on your cluster environment and operating system. If it's a managed cluster, you may need admin access or an scrip...

  • 2 kudos
15 More Replies
batch_bender
by New Contributor II
  • 109 Views
  • 2 replies
  • 1 kudos

Folder structure for Autoloader and Declarative Pipelines

We have six years of Parquet files organised under yyyy/mm/dd/, with around 100 table files in each daily folder. We want to process these 100 tables using Auto Loader in a Declarative Pipeline with the AUTO CDC API.If each table reads the same root ...

  • 109 Views
  • 2 replies
  • 1 kudos
Latest Reply
GabFernandes
New Contributor III
  • 1 kudos

Hi @batch_bender ,Yes, pointing Auto Loader to the root path with a filename wildcard for every table will cause significant file discovery overhead, as each stream independently crawls the entire directory tree (yyyy/mm/dd/).Best Practice Recommenda...

  • 1 kudos
1 More Replies
matko
by Visitor
  • 72 Views
  • 1 replies
  • 0 kudos

oreign Iceberg tables via HMS federation fail with browse_only_table on all compute — request Public

Per your documentation (hms-federation-concepts), Apache Iceberg tables in federated external Hive metastores are Public Preview on Databricks Runtime 16.2+, with no enablement steps listed. On our workspace, every read of a federated Iceberg table f...

  • 72 Views
  • 1 replies
  • 0 kudos
Latest Reply
balajij8
Esteemed Contributor
  • 0 kudos

Hi matko,The underlying issue here is mostly a missing backend feature enablement for the workspace. Databricks frequently gates these newer federation capabilities behind workspace-level or account-level enablement during the preview phase.Unity Cat...

  • 0 kudos
Sam500
by New Contributor III
  • 77 Views
  • 1 replies
  • 0 kudos

Multi-Engine Lakehouses

Hi experts,I recently came across an interesting discussion titled “Are enterprises moving from ‘Data Lakehouse’ to ‘Agentic Lakehouse’?” https://community.databricks.com/t5/data-engineering/are-enterprises-moving-from-quot-data-lakehouse-quot-to-quo...

  • 77 Views
  • 1 replies
  • 0 kudos
Latest Reply
balajij8
Esteemed Contributor
  • 0 kudos

Hi Sam500,The multi engine pattern usually comes to establishing interoperability at the storage format level while maintaining a single, unified governance layer. You handle this primarily through Unity Catalog (UC) and format flexibility such as De...

  • 0 kudos
binlogreader
by New Contributor II
  • 84 Views
  • 2 replies
  • 1 kudos

DLT driver GC pressure during a large initial hydration. Is a bigger driver the only lever?

Last week one of our DLT pipelines spent five hours in RUNNING with nothing landing, and I'd like to compare notes on how people handle the underlying problem, because we found one fix and several dead ends.Setup: A continuous DLT pipeline on classic...

  • 84 Views
  • 2 replies
  • 1 kudos
Latest Reply
masonreed11
New Contributor II
  • 1 kudos

I ran into something similar, and the event log was the only place that made the root cause obvious. I don't believe there's an officially supported setting to limit concurrent DLT flows, so I'd avoid undocumented configs. For large initial backfills...

  • 1 kudos
1 More Replies
Dhruv-22
by Contributor III
  • 242 Views
  • 6 replies
  • 0 kudos

VOID column inside STRUCT fails to cast to VARIANT

Let's say I have the following dataframe%sqlCREATE OR REPLACE TEMPORARY VIEW v_temp ASVALUES (NULL, DATE'2025-12-31'), (NULL, DATE'2026-04-12'), (NULL, DATE'2026-06-23') AS T(PAYMENT_DATE, TRANSACTION_DATE);SELECT PAYMENT_DATE, TYPEOF(PAYMENT_DATE), ...

Dhruv22_0-1785136495855.png Dhruv22_1-1785136581445.png Dhruv22_2-1785136658557.png Dhruv22_3-1785136718467.png
  • 242 Views
  • 6 replies
  • 0 kudos
Latest Reply
anagilla
Databricks Employee
  • 0 kudos

Your screenshot already uses TO_VARIANT_OBJECT; the VOID field inside the STRUCT is what causes the failure. TO_VARIANT_OBJECT requires every field to be convertible to VARIANT, but VOID is not accepted in that nested path. Scalar CAST(PAYMENT_DATE A...

  • 0 kudos
5 More Replies
swang-pg
by New Contributor II
  • 91 Views
  • 1 replies
  • 2 kudos

Merging 2 versions of an SCD table created from a managed ingestion pipeline

We've been using the Salesforce Databricks connector to ingest data into Databricks with history tracking (SCD2) turned on. The licence to one of the add-on modules was accidentally revoked for our ingestion user and subsequently reinstated. This led...

  • 91 Views
  • 1 replies
  • 2 kudos
Latest Reply
binlogreader
New Contributor II
  • 2 kudos

@swang-pg We did not run into the error but had a similar situation of stitching tables while using Salesforce and Mysql connectors so I can share what we did. Two things about the situation shape what the merge can look like.The first is that the me...

  • 2 kudos
Labels