cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

ChristianRRL
by Honored Contributor II
  • 40 Views
  • 1 replies
  • 0 kudos

CREATE CONNECTION - Support for Community Connections?

Hi there, as the title asks, I'm wondering how Databricks Community Connections are being supported. For example, when I try to make a fresh connection via Spark SQL commands, I get an error because `COMMUNITY` type is not supported.However, when I c...

ChristianRRL_0-1785276761138.png ChristianRRL_1-1785276787254.png ChristianRRL_4-1785277045154.png ChristianRRL_3-1785276855186.png
  • 40 Views
  • 1 replies
  • 0 kudos
Latest Reply
adnan_alvee
Databricks Employee
  • 0 kudos

Lakeflow Community Connector connections are supported, but not as a normal Spark SQL CREATE CONNECTION type today. They’re supported through the Lakeflow Community Connectors flow/UI and companion CLI/API path, while SQL/SDK enum surfaces still don’...

  • 0 kudos
shama1
by New Contributor II
  • 4276 Views
  • 3 replies
  • 3 kudos

Resolved! Not able to reset password with password re-set link

I am not able to reset password https://community.cloud.databricks.com/When I use link provided in the email to re-set the password, and I enter new password, request never gets completed.Password reset link keeps on loading even after entering passw...

image
  • 4276 Views
  • 3 replies
  • 3 kudos
Latest Reply
Anusha6
Visitor
  • 3 kudos

I am not able to connect to Databricks partner academy login. The databricks SSO login directs me to MFA passkey. But I donot have a passkey setup.

  • 3 kudos
2 More Replies
maninegi05
by New Contributor II
  • 142 Views
  • 1 replies
  • 0 kudos

Databricks Apps (Streamlit) - How to Implement Proper Logout Functionality?

We have built a Databricks App using Streamlit and are looking for a way to implement a proper logout functionality.Currently, we built Logout functionality where we can clear the Streamlit session state and client-side cookies. However, the Databric...

  • 142 Views
  • 1 replies
  • 0 kudos
Latest Reply
emma_s
Databricks Employee
  • 0 kudos

Hi, At the moment this is not a supported feature, you may want to raise it as a feautre request via your account team. Having said that there are a few things you could try to get a near but not perfect experience. Here are some options for you to e...

  • 0 kudos
LincolnGarbin
by New Contributor
  • 140 Views
  • 2 replies
  • 3 kudos

Resolved! What happens to Databricks notebooks and jobs when a user's Microsoft account is deactivated?

Hi everyone,I recently joined a company where we use Databricks to process and transform data that is ultimately used in our Power BI reports and dashboards.The person who has been training me is leaving the company. Our Databricks environment is int...

  • 140 Views
  • 2 replies
  • 3 kudos
Latest Reply
GabFernandes
New Contributor III
  • 3 kudos

Hi!This is a very common scenario when dealing with identity integration between Microsoft Entra ID (Azure AD) and Databricks.Here is a breakdown of what happens when a corporate account is deactivated, followed by the best practices to ensure zero d...

  • 3 kudos
1 More Replies
Dhruv-22
by Contributor III
  • 142 Views
  • 3 replies
  • 0 kudos

VOID column inside STRUCT fails to cast to VARIANT

Let's say I have the following dataframe%sqlCREATE OR REPLACE TEMPORARY VIEW v_temp ASVALUES (NULL, DATE'2025-12-31'), (NULL, DATE'2026-04-12'), (NULL, DATE'2026-06-23') AS T(PAYMENT_DATE, TRANSACTION_DATE);SELECT PAYMENT_DATE, TYPEOF(PAYMENT_DATE), ...

Dhruv22_0-1785136495855.png Dhruv22_1-1785136581445.png Dhruv22_2-1785136658557.png Dhruv22_3-1785136718467.png
  • 142 Views
  • 3 replies
  • 0 kudos
Latest Reply
balajij8
Esteemed Contributor
  • 0 kudos

Hi Dhruv,The cast to VARIANT generally fails because complex types (like STRUCT, ARRAY or MAP) require strict, resolvable data types for serialization. When a column is entirely NULL and implicitly defined, Spark infers it as VOID which lacks the rep...

  • 0 kudos
2 More Replies
droid
by New Contributor II
  • 80 Views
  • 0 replies
  • 0 kudos

Support Multiple Tasks DAG Inside a `for_each_task` Iteration

Databricks Jobs `for_each_task` accepts exactly one nested task. For workloads that run the *samemulti-step pipeline over many datasets* — a very common metadata-driven pattern — this forces achoice between two architectures, each of which gives up s...

  • 80 Views
  • 0 replies
  • 0 kudos
17abhishek
by New Contributor III
  • 109 Views
  • 1 replies
  • 0 kudos

Issue with Notebook Scrolling Position While Switching Between Multiple Notebooks

Hi Team,For the past 1–2 weeks, I have been experiencing an issue while working with multiple notebooks simultaneously. Whenever I switch from one notebook to another, the notebook automatically scrolls back to Cell 1 instead of retaining my current ...

  • 109 Views
  • 1 replies
  • 0 kudos
Latest Reply
Brahmareddy
Esteemed Contributor II
  • 0 kudos

Hi Abhi,How are you doing today? as per my understanding, this looks like a notebook state or browser session issue. As a temporary workaround, try opening notebooks in separate browser tabs, testing in incognito mode, or clearing the browser cache.I...

  • 0 kudos
AlexSantiago
by New Contributor II
  • 19781 Views
  • 29 replies
  • 4 kudos

spotify API get token - raw_input was called, but this frontend does not support input requests.

hello everyone, I'm trying use spotify's api to analyse my music data, but i'm receiving a error during authentication, specifically when I try get the token, above my code.Is it a databricks bug?pip install spotipyfrom spotipy.oauth2 import SpotifyO...

  • 19781 Views
  • 29 replies
  • 4 kudos
Latest Reply
delta432
New Contributor
  • 4 kudos

I run Delta Xecutor a website dedicated to providing the latest information about Delta Executor for Roblox. We publish installation guides, download tutorials, update news, FAQs, troubleshooting tips, and beginner-friendly walkthroughs to help users...

  • 4 kudos
28 More Replies
maddan80
by New Contributor III
  • 4164 Views
  • 8 replies
  • 4 kudos

Oracle Essbase connectivity

Team, I wanted to understand the best way of connecting to Oracle Essbase to ingest data into the delta lake

  • 4164 Views
  • 8 replies
  • 4 kudos
Latest Reply
binlogreader
New Contributor II
  • 4 kudos

@hyaqoob @saf_dbx Your query and your connection are both fine. Error 10220 is the Databricks JDBC driver's SQLFeatureNotSupportedException, which means the tool called an optional JDBC API method the driver does not implement. It happens during data...

  • 4 kudos
7 More Replies
van45678
by New Contributor III
  • 189 Views
  • 2 replies
  • 2 kudos

Auto-Termination Did Not Trigger on Production Cluster Despite 20-Minute Inactivity

Hello everyone,I recently encountered an issue with one of our production all-purpose clusters where the auto-termination feature did not work as expected.The cluster was configured with "Terminate after 20 minutes of inactivity", but it never termin...

van45678_0-1784894552797.png van45678_1-1784894588839.png
  • 189 Views
  • 2 replies
  • 2 kudos
Latest Reply
binlogreader
New Contributor II
  • 2 kudos

@van45678 -- @balajij8 's diagnosis matches what your Spark UI shows, so I will just add three things from the operational side, one for the postmortem and two for making sure the next occurrence costs you hours instead of five days.(1) Driver logs a...

  • 2 kudos
1 More Replies
Areqio
by New Contributor II
  • 203 Views
  • 2 replies
  • 1 kudos

can i deploy a metric view using DABs

I am trying to deploy metric view using DABs but I cant find a resource for it so i ended up with a SQL file but is there a way to automatically run the SQL for it

  • 203 Views
  • 2 replies
  • 1 kudos
Latest Reply
binlogreader
New Contributor II
  • 1 kudos

@Areqio there is no metric view resource in Asset Bundles today. Your SQL file is the intended shape for anything the bundle doesn't model yet. The missing piece is a job that runs it, so deployment becomes deploy then run.You can with just two scrip...

  • 1 kudos
1 More Replies
Bitrip007
by New Contributor II
  • 223 Views
  • 1 replies
  • 1 kudos

Are enterprises moving from "Data Lakehouse" to "Agentic Lakehouse"?

Introduction several capabilities that seem to shift Databricks from being primarily a data platform to becoming an AI operating system—Genie One, Agent Bricks, AI Gateway, Lakebase Search, and business semantics.Do you see the future enterprise arch...

  • 223 Views
  • 1 replies
  • 1 kudos
Latest Reply
suryaprayaga
New Contributor II
  • 1 kudos

Yes its true. I work in an AdTech and MarTech company where we serve customers with rich AdTech use cases like Campaign Optimization, Creative Performance, AdOps etc., In the recent past we have seen multiple use cases with several customers asking f...

  • 1 kudos
raman7292
by New Contributor
  • 218 Views
  • 1 replies
  • 1 kudos

Use of Genie agents for ETL migration

I am currently working on a project to migrate existing ETL workflows from SSIS to Databricks. I would like to understand how Databricks Genie, Genie Code, or other Databricks AI-assisted capabilities can support this migration.Can these tools analyz...

  • 218 Views
  • 1 replies
  • 1 kudos
Latest Reply
adnan_alvee
Databricks Employee
  • 1 kudos

Genie code can meaningfully accelerate an SSIS migration, but they should be viewed as a productivity layer around the migration process rather than a push-button solution. In practice, the most reliable automation today is around assessment, invento...

  • 1 kudos
ChristianRRL
by Honored Contributor II
  • 272 Views
  • 2 replies
  • 1 kudos

Resolved! Community Custom Connector - Defining Non-Serverless Job Compute Runtime?

Hi there,I'm exploring Community Custom Connectors and I'm wondering about the Compute. By default, it's set to Serverless:But due to IT network limitations at our company, we cannot use serverless. In the Compute settings, I can untoggle the Serverl...

ChristianRRL_0-1784905334067.png ChristianRRL_1-1784905449577.png
  • 272 Views
  • 2 replies
  • 1 kudos
Latest Reply
adnan_alvee
Databricks Employee
  • 1 kudos

You generally cannot select DBR version in Lakeflow / Spark Declarative Pipelines because the pipeline runtime is Databricks-managed and versionless. Databricks auto-upgrades the runtime for pipelines.The settings has a JSON view, and some advanced o...

  • 1 kudos
1 More Replies
Labels