cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

AlexSantiago
by New Contributor II
  • 19761 Views
  • 29 replies
  • 4 kudos

spotify API get token - raw_input was called, but this frontend does not support input requests.

hello everyone, I'm trying use spotify's api to analyse my music data, but i'm receiving a error during authentication, specifically when I try get the token, above my code.Is it a databricks bug?pip install spotipyfrom spotipy.oauth2 import SpotifyO...

  • 19761 Views
  • 29 replies
  • 4 kudos
Latest Reply
delta432
Visitor
  • 4 kudos

I run Delta Xecutor a website dedicated to providing the latest information about Delta Executor for Roblox. We publish installation guides, download tutorials, update news, FAQs, troubleshooting tips, and beginner-friendly walkthroughs to help users...

  • 4 kudos
28 More Replies
Dhruv-22
by Contributor III
  • 45 Views
  • 0 replies
  • 0 kudos

VOID column inside STRUCT fails to cast to VARIANT

Let's say I have the following dataframe%sqlCREATE OR REPLACE TEMPORARY VIEW v_temp ASVALUES (NULL, DATE'2025-12-31'), (NULL, DATE'2026-04-12'), (NULL, DATE'2026-06-23') AS T(PAYMENT_DATE, TRANSACTION_DATE);SELECT PAYMENT_DATE, TYPEOF(PAYMENT_DATE), ...

Dhruv22_0-1785136495855.png Dhruv22_1-1785136581445.png Dhruv22_2-1785136658557.png Dhruv22_3-1785136718467.png
  • 45 Views
  • 0 replies
  • 0 kudos
maddan80
by New Contributor III
  • 4139 Views
  • 8 replies
  • 4 kudos

Oracle Essbase connectivity

Team, I wanted to understand the best way of connecting to Oracle Essbase to ingest data into the delta lake

  • 4139 Views
  • 8 replies
  • 4 kudos
Latest Reply
binlogreader
New Contributor II
  • 4 kudos

@hyaqoob @saf_dbx Your query and your connection are both fine. Error 10220 is the Databricks JDBC driver's SQLFeatureNotSupportedException, which means the tool called an optional JDBC API method the driver does not implement. It happens during data...

  • 4 kudos
7 More Replies
van45678
by New Contributor III
  • 174 Views
  • 2 replies
  • 2 kudos

Auto-Termination Did Not Trigger on Production Cluster Despite 20-Minute Inactivity

Hello everyone,I recently encountered an issue with one of our production all-purpose clusters where the auto-termination feature did not work as expected.The cluster was configured with "Terminate after 20 minutes of inactivity", but it never termin...

van45678_0-1784894552797.png van45678_1-1784894588839.png
  • 174 Views
  • 2 replies
  • 2 kudos
Latest Reply
binlogreader
New Contributor II
  • 2 kudos

@van45678 -- @balajij8 's diagnosis matches what your Spark UI shows, so I will just add three things from the operational side, one for the postmortem and two for making sure the next occurrence costs you hours instead of five days.(1) Driver logs a...

  • 2 kudos
1 More Replies
Areqio
by New Contributor II
  • 190 Views
  • 2 replies
  • 1 kudos

can i deploy a metric view using DABs

I am trying to deploy metric view using DABs but I cant find a resource for it so i ended up with a SQL file but is there a way to automatically run the SQL for it

  • 190 Views
  • 2 replies
  • 1 kudos
Latest Reply
binlogreader
New Contributor II
  • 1 kudos

@Areqio there is no metric view resource in Asset Bundles today. Your SQL file is the intended shape for anything the bundle doesn't model yet. The missing piece is a job that runs it, so deployment becomes deploy then run.You can with just two scrip...

  • 1 kudos
1 More Replies
Bitrip007
by New Contributor II
  • 211 Views
  • 1 replies
  • 1 kudos

Are enterprises moving from "Data Lakehouse" to "Agentic Lakehouse"?

Introduction several capabilities that seem to shift Databricks from being primarily a data platform to becoming an AI operating system—Genie One, Agent Bricks, AI Gateway, Lakebase Search, and business semantics.Do you see the future enterprise arch...

  • 211 Views
  • 1 replies
  • 1 kudos
Latest Reply
suryaprayaga
New Contributor II
  • 1 kudos

Yes its true. I work in an AdTech and MarTech company where we serve customers with rich AdTech use cases like Campaign Optimization, Creative Performance, AdOps etc., In the recent past we have seen multiple use cases with several customers asking f...

  • 1 kudos
Brahmareddy
by Esteemed Contributor II
  • 175 Views
  • 1 replies
  • 2 kudos

Reducing Time to Help in the Databricks Community

Hi @MandyR , @Advika , and teamI wanted to share one idea for the Databricks Community. Can we explore a Genie-like experience for databricks community?The Community already has many useful answers, accepted solutions, blogs, MVP articles, and discus...

  • 175 Views
  • 1 replies
  • 2 kudos
Latest Reply
suryaprayaga
New Contributor II
  • 2 kudos

Yes totally agree with this. Posts can be sub-categorized, tags can be created and responses can be scored based on how effective or important they are (even though this is already a feature currently).I did a similar exercise for Amazon earlier wher...

  • 2 kudos
raman7292
by New Contributor
  • 190 Views
  • 1 replies
  • 1 kudos

Use of Genie agents for ETL migration

I am currently working on a project to migrate existing ETL workflows from SSIS to Databricks. I would like to understand how Databricks Genie, Genie Code, or other Databricks AI-assisted capabilities can support this migration.Can these tools analyz...

  • 190 Views
  • 1 replies
  • 1 kudos
Latest Reply
adnan_alvee
Databricks Employee
  • 1 kudos

Genie code can meaningfully accelerate an SSIS migration, but they should be viewed as a productivity layer around the migration process rather than a push-button solution. In practice, the most reliable automation today is around assessment, invento...

  • 1 kudos
ChristianRRL
by Honored Contributor
  • 233 Views
  • 2 replies
  • 1 kudos

Resolved! Community Custom Connector - Defining Non-Serverless Job Compute Runtime?

Hi there,I'm exploring Community Custom Connectors and I'm wondering about the Compute. By default, it's set to Serverless:But due to IT network limitations at our company, we cannot use serverless. In the Compute settings, I can untoggle the Serverl...

ChristianRRL_0-1784905334067.png ChristianRRL_1-1784905449577.png
  • 233 Views
  • 2 replies
  • 1 kudos
Latest Reply
adnan_alvee
Databricks Employee
  • 1 kudos

You generally cannot select DBR version in Lakeflow / Spark Declarative Pipelines because the pipeline runtime is Databricks-managed and versionless. Databricks auto-upgrades the runtime for pipelines.The settings has a JSON view, and some advanced o...

  • 1 kudos
1 More Replies
van45678
by New Contributor III
  • 6361 Views
  • 3 replies
  • 0 kudos

Resolved! Getting connection reset issue while connecting to a SQL server

Hello All,I am unable to connect to a SQL server instance that is installed in a on-premise network from databricks. I am able to successfully ping the server from the notebook using this command [nc -vz <hostname> <port>]  which means I am able to e...

Data Engineering
Databricks
sqlserver
timeout
  • 6361 Views
  • 3 replies
  • 0 kudos
Latest Reply
mark_ott
Databricks Employee
  • 0 kudos

The error you are encountering, "com.microsoft.sqlserver.jdbc.SQLServerException: Connection reset," even after a successful nc (netcat) connection, is a common but nuanced problem when connecting Databricks to an on-premise SQL Server. Although your...

  • 0 kudos
2 More Replies
Abarghodrat
by New Contributor II
  • 377 Views
  • 6 replies
  • 1 kudos

My Apps get stopped automatically

Hi team,I have 2 applications in my workspace which they will get stopped sometimes without me knowing the reason. I’m on premium account and I have to run them when I can’t have access to my app. I need help. â€ƒâ€ƒâ€ƒ

IMG_7803.png IMG_7790.png
  • 377 Views
  • 6 replies
  • 1 kudos
Latest Reply
DataNexus
New Contributor
  • 1 kudos

Could you check the app Logs and tell me what you see when one of them stops?A stack trace or "OOM / killed" → it's crashing (code/memory fix)Clean shutdown with no error → it's the compute lifecycle (keep-alive job fix)Nothing / app never fully star...

  • 1 kudos
5 More Replies
binlogreader
by New Contributor II
  • 423 Views
  • 4 replies
  • 2 kudos

Resolved! sdp-meta (dlt-meta) vs lakeflow_framework: when should we use which?

Hi All,I'm evaluating metadata-driven frameworks for Lakeflow Spark Declarative Pipelines, and I've found two that appear to serve the same purpose, both from Databricks organizations:- databrickslabs/dlt-meta, whose feature/sdp-meta branch renames t...

  • 423 Views
  • 4 replies
  • 2 kudos
Latest Reply
anagilla
Databricks Employee
  • 2 kudos

Since you are already on Lakeflow Connect for MySQL, your CDC ingestion into Bronze is the managed, supported path, so you do not need a metadata framework for it. Resources by layer: Ingestion to Bronze (what you have): stay on Lakeflow Connect. It...

  • 2 kudos
3 More Replies
ChristianRRL
by Honored Contributor
  • 336 Views
  • 2 replies
  • 2 kudos

Resolved! What is the compute for Lakeflow Connect SharePoint Connector

Can someone help me understand what compute the SharePoint Connector is using by default? I'm trying to use Job Compute, but it "seems" like it may be defaulting to serverless. 

ChristianRRL_0-1784588867082.png
  • 336 Views
  • 2 replies
  • 2 kudos
Latest Reply
anagilla
Databricks Employee
  • 2 kudos

The managed SharePoint connector uses serverless compute; it can't be switched to classic Job Compute. The Job you see is the orchestration layer: Lakeflow Connect creates it for the schedule and runs the ingestion pipeline as a task, while the pipel...

  • 2 kudos
1 More Replies
Labels