Hello Databricks Community,I recently published the V.E.N.K.A.T Framework™, an enterprise architecture model for the Agentic AI era.The idea is simple:As AI moves from generating insights to taking actions, the architecture underneath becomes critica...
Hi,I am not a Data Engineer, I want to connect to ssas. It looks like it can be connected through pyodbc. however looks like I need to install "ODBC Driver 17 for SQL Server" using the following command. How do i install the driver on the cluster an...
If you're only connecting to SSAS, installing the ODBC Driver 17 for SQL Server is usually the first step, but the exact method depends on your cluster environment and operating system. If it's a managed cluster, you may need admin access or an scrip...
We have six years of Parquet files organised under yyyy/mm/dd/, with around 100 table files in each daily folder. We want to process these 100 tables using Auto Loader in a Declarative Pipeline with the AUTO CDC API.If each table reads the same root ...
Hi @batch_bender ,Yes, pointing Auto Loader to the root path with a filename wildcard for every table will cause significant file discovery overhead, as each stream independently crawls the entire directory tree (yyyy/mm/dd/).Best Practice Recommenda...
Per your documentation (hms-federation-concepts), Apache Iceberg tables in federated external Hive metastores are Public Preview on Databricks Runtime 16.2+, with no enablement steps listed. On our workspace, every read of a federated Iceberg table f...
Hi matko,The underlying issue here is mostly a missing backend feature enablement for the workspace. Databricks frequently gates these newer federation capabilities behind workspace-level or account-level enablement during the preview phase.Unity Cat...
Hi experts,I recently came across an interesting discussion titled “Are enterprises moving from ‘Data Lakehouse’ to ‘Agentic Lakehouse’?” https://community.databricks.com/t5/data-engineering/are-enterprises-moving-from-quot-data-lakehouse-quot-to-quo...
Hi Sam500,The multi engine pattern usually comes to establishing interoperability at the storage format level while maintaining a single, unified governance layer. You handle this primarily through Unity Catalog (UC) and format flexibility such as De...
Last week one of our DLT pipelines spent five hours in RUNNING with nothing landing, and I'd like to compare notes on how people handle the underlying problem, because we found one fix and several dead ends.Setup: A continuous DLT pipeline on classic...
I ran into something similar, and the event log was the only place that made the root cause obvious. I don't believe there's an officially supported setting to limit concurrent DLT flows, so I'd avoid undocumented configs. For large initial backfills...
Let's say I have the following dataframe%sqlCREATE OR REPLACE TEMPORARY VIEW v_temp ASVALUES (NULL, DATE'2025-12-31'), (NULL, DATE'2026-04-12'), (NULL, DATE'2026-06-23') AS T(PAYMENT_DATE, TRANSACTION_DATE);SELECT PAYMENT_DATE, TYPEOF(PAYMENT_DATE), ...
Your screenshot already uses TO_VARIANT_OBJECT; the VOID field inside the STRUCT is what causes the failure. TO_VARIANT_OBJECT requires every field to be convertible to VARIANT, but VOID is not accepted in that nested path. Scalar CAST(PAYMENT_DATE A...
We've been using the Salesforce Databricks connector to ingest data into Databricks with history tracking (SCD2) turned on. The licence to one of the add-on modules was accidentally revoked for our ingestion user and subsequently reinstated. This led...
@swang-pg We did not run into the error but had a similar situation of stitching tables while using Salesforce and Mysql connectors so I can share what we did. Two things about the situation shape what the merge can look like.The first is that the me...
Hi Community,I am currently using the Free Edition and I would like to explore Genie Ontology as part of a project.I checked my account and it seems Genie Ontology is not yet available on the Free Edition. Could someone guide me on how to request ac...
Hi!To answer your question directly: Databricks Genie (and Genie Spaces / Ontology) is not available on the Databricks Community Edition (Free Edition), and there is currently no waitlist to enable it within that environment.Here is why, along with t...
Hi Everyone, There is a need for a lakehouse front-end tool at my company that operates like an excel spreadsheet but reads/writes to a lakehouse table. This would replace actual spreadsheets that are passed around, manually updated, and then collate...
Hi Team,How can I set up multiple languages in Databricks? For example, if I connect from Germany, the workspace and data should support German. If I connect from China, it should support Chinese, and if I connect from the US, it should be in English...
Great question! Databricks doesn't automatically change the workspace or data language based on a user's location. The best practice is to keep your data language-neutral, store translations in separate tables, detect the user's preferred language (s...
I am not able to reset password https://community.cloud.databricks.com/When I use link provided in the email to re-set the password, and I enter new password, request never gets completed.Password reset link keeps on loading even after entering passw...
I am not able to connect to Databricks partner academy login. The databricks SSO login directs me to MFA passkey. But I donot have a passkey setup.
We have built a Databricks App using Streamlit and are looking for a way to implement a proper logout functionality.Currently, we built Logout functionality where we can clear the Streamlit session state and client-side cookies. However, the Databric...
Hi, At the moment this is not a supported feature, you may want to raise it as a feautre request via your account team. Having said that there are a few things you could try to get a near but not perfect experience. Here are some options for you to e...
Hi everyone,I recently joined a company where we use Databricks to process and transform data that is ultimately used in our Power BI reports and dashboards.The person who has been training me is leaving the company. Our Databricks environment is int...
Hi!This is a very common scenario when dealing with identity integration between Microsoft Entra ID (Azure AD) and Databricks.Here is a breakdown of what happens when a corporate account is deactivated, followed by the best practices to ensure zero d...
Hi Team,For the past 1–2 weeks, I have been experiencing an issue while working with multiple notebooks simultaneously. Whenever I switch from one notebook to another, the notebook automatically scrolls back to Cell 1 instead of retaining my current ...
Hi Abhi,How are you doing today? as per my understanding, this looks like a notebook state or browser session issue. As a temporary workaround, try opening notebooks in separate browser tabs, testing in incognito mode, or clearing the browser cache.I...
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 586 | |
| 349 | |
| 287 |
