cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

Start a conversation
AlexSantiago
by • New Contributor
  • 1263 Views
  • 5 replies
  • 0 kudos

spotify API get token - raw_input was called, but this frontend does not support input requests.

hello everyone, I'm trying use spotify's api to analyse my music data, but i'm receiving a error during authentication, specifically when I try get the token, above my code.Is it a databricks bug?pip install spotipyfrom spotipy.oauth2 import SpotifyO...

Data Engineering
Reply
  • 1263 Views
  • 5 replies
  • 0 kudos
View Replies
Latest Reply
SabtainSaleem
New Contributor II
  • 0 kudos

When working with the Spotify API and dealing with authentication issues, especially around token generation, it's important to ensure you're following the correct OAuth flow. Double-check your backend setup for token handling and make sure the raw i...

  • 0 kudos
4 More Replies
ak4
by • New Contributor
  • 93 Views
  • 1 replies
  • 0 kudos

Failed to read job commit marker error

Recently, we migrate from DBR 11.3 LTS ML to DBR 14.3 LTS ML. We are struggling on one data source where we consume parquet files. New data are appended every 30 minutes to that data source. The data are generated by Databricks notebook which runs on...

Data Engineering
Reply
  • 93 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
menotron
Valued Contributor
  • 0 kudos

Hi @ak4, this is an expected behavior with DBIO transactional commit enabled and most likely the issue is when you update a table and query it immediately. You could explicitly invalidate the cache or configure disk cache. If you have a long running ...

  • 0 kudos
simple89
by • Visitor
  • 19 Views
  • 0 replies
  • 0 kudos

Runtime increases exponentially from 11.3 to 13.3

Hello. I am using R on databricks and using the below approach. My Spark version:Single node: i3.2xlarge · On-demand · DBR: 11.3 LTS (includes Apache Spark 3.3.0, Scala 2.12) · us-east-1a, the job takes 1 hourI install all R packages (including a geo...

Data Engineering
Reply
  • 19 Views
  • 0 replies
  • 0 kudos
ae20cg
by • New Contributor III
  • 3020 Views
  • 5 replies
  • 9 kudos

Databricks Cluster Web terminal different permissions with tmux and xterm.

I am launching web terminal on my databricks cluster and when I am using the ephemeral xterm instance I am easily able to navigate to desired directory in `Workspace` and run anything... for example `ls ./` When I switch to tmux so that I can preserv...

Data Engineering
Reply
  • 3020 Views
  • 5 replies
  • 9 kudos
View Replies
Latest Reply
alenka
New Contributor III
  • 9 kudos

Hey there, fellow data explorer pals! I totally get your excitement when launching that web terminal on your Databricks cluster and feeling the power of running commands like 'ls ./' in the ephemeral xterm instance. It's like traversing the vast univ...

  • 9 kudos
4 More Replies
sandy311
by • New Contributor
  • 35 Views
  • 1 replies
  • 0 kudos

Databricks asset bundle does not create new job if I change configuration of existing Databricks yam

When deploying multiple jobs using the `Databricks.yml` file via the asset bundle, the process either overwrites the same job or renames it, instead of creating separate, distinct jobs.

Data Engineering
Reply
  • 35 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
filipniziol
New Contributor II
  • 0 kudos

Hi @sandy311 ,could you share your databricks.yml file?Are you sure you used unique job ids when defining your jobs?  

  • 0 kudos
kranthi2
by • New Contributor II
  • 42 Views
  • 2 replies
  • 1 kudos

Resolved! alter DLT Materialized View alter column set MASK

I am trying to mask a column on a DLT materialized view - this is created using DLT syntax. I am not able set the column masking after creation. Appreciate any workaround.alter DLT Materialized View alter column set MASK

Data Engineering
Reply
  • 42 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
kranthi2
New Contributor II
  • 1 kudos

Thank you. I will submit the idea.

  • 1 kudos
1 More Replies
Nis
by • New Contributor II
  • 1472 Views
  • 2 replies
  • 1 kudos

can we commit offset in spark structured streaming in databricks.

We are storing offset details in checkpoint location wanted to know is there a way can we commit offset once we consume the message from kafka.

Data Engineering
Reply
  • 1472 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
dmytro
New Contributor II
  • 1 kudos

Hi @raphaelblg , thanks a lot for providing an elaborate answer. Do you happen to you, by any chance, of some solutions that developers use to track a consumer lag when streaming with Spark from a Kafka topic? It's a rather essential knowledge to hav...

  • 1 kudos
1 More Replies
prasadvaze
by • Valued Contributor II
  • 15960 Views
  • 15 replies
  • 12 kudos

Resolved! How to query delta lake using SQL desktop tools like SSMS or DBVisualizer

Is there a way to use sql desktop tools? because delta OSS or databricks does not provide desktop client (similar to azure data studio) to browse and query delta lake objects.I currently use databricks SQL , a webUI in the databricks workspace but se...

Data Engineering
Reply
  • 15960 Views
  • 15 replies
  • 12 kudos
View Replies
Latest Reply
prasadvaze
Valued Contributor II
  • 12 kudos

DSR is Delta Standalone Reader. see more here - https://docs.delta.io/latest/delta-standalone.htmlIts a crate (and also now a py library) that allows you to connect to delta tables without using spark (e.g. directly from python and not using pyspa...

  • 12 kudos
14 More Replies
pragarwal
by • New Contributor II
  • 1232 Views
  • 6 replies
  • 0 kudos

Adding Member to group using account databricks rest api

Hi All,I want to add a member to a group in databricks account level using rest api (https://docs.databricks.com/api/azure/account/accountgroups/patch) as mentioned in this link I could able to authenticate but not able to add member while using belo...

Data Engineering
Reply
  • 1232 Views
  • 6 replies
  • 0 kudos
View Replies
Latest Reply
Nikos
New Contributor
  • 0 kudos

Does the above work? I still can't quite figure it out. Any help would be much appreciated.I know authentication is not an issue as I can use a lot of the other endpoints. I just can't figure out the correct body syntax to add a member to a group.url...

  • 0 kudos
5 More Replies
oleh_v
by • New Contributor
  • 91 Views
  • 2 replies
  • 0 kudos

Upload of .bin file >400mb

I try to upload to local workspace folder with .bin extension.It is required to have it locally.I tried load from DBFS, but loading files over 265mb is not allowed with cluster. I tried to upload manually but failed with same error "OSError: [Errno5]...

Data Engineering
Reply
  • 91 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
Kartheek_Katta
New Contributor
  • 0 kudos

Hello Slash,Thank you for your response. I'm encountering the same issue as described. I tried running the provided code in my Databricks workspace, but I received an error. My question is how the script is expected to access local files, especially ...

  • 0 kudos
1 More Replies
sakuraDev
by • New Contributor
  • 51 Views
  • 0 replies
  • 0 kudos

Why does soda not initialize?

Hey everyone, im using autoloader x soda.I'm new to both,The idea is to ingest with quality checks in my silver table for every batch in a continuous ingestion.I tried to configure soda as str just like the docs show, but its seems that it keeps on t...

sakuraDev_0-1725645131588.png
Data Engineering
Reply
  • 51 Views
  • 0 replies
  • 0 kudos
cm04
by • New Contributor II
  • 28 Views
  • 0 replies
  • 0 kudos

Why does my job run on shared compute instead of job compute?

I have configured a job using `databricks.yml````resources:  jobs:    my_job:      name: my_job      tasks:        - task_key: create_feature_tables          job_cluster_key: my_job_cluster          spark_python_task:            python_file: ../src/c...

cm04_0-1725643451954.png
Data Engineering
Reply
  • 28 Views
  • 0 replies
  • 0 kudos
Meghana_Vasavad
by • New Contributor II
  • 187 Views
  • 4 replies
  • 0 kudos

Resolved! Discrepancies in Data Engineering GitHub Repositories and Errors in Databricks Notebooks

Hi team,I am writing to express my concerns regarding the recent Databricks webinar on "Data Intelligence with Databricks". During the session, there was mention of two GitHub repositories associated with the notebooks being used as mentioned below.1...

Data Engineering
Reply
  • 187 Views
  • 4 replies
  • 0 kudos
View Replies
Latest Reply
Sujitha
Community Manager
  • 0 kudos

Hi @Meghana_Vasavad could you please file a ticket with Databricks SupportThey will help you with this request. 

  • 0 kudos
3 More Replies
raghu2
by • New Contributor III
  • 1547 Views
  • 5 replies
  • 0 kudos

Resolved! Error deploying a DAB

I followed steps listed in this article.After creating and validation of bundle with default template, during deployment using this command:databricks bundle deploy -t dev --profile zzI get this message:Building mySecPrj...Error: build failed mySecPr...

Data Engineering
DAB
Reply
  • 1547 Views
  • 5 replies
  • 0 kudos
View Replies
Latest Reply
ADB0513
New Contributor III
  • 0 kudos

@daniel_sahal I am receiving the same error and tried your solution and am still getting the invalid command bdist_wheel.  Any other suggestions?  Thanks

  • 0 kudos
4 More Replies
rendorHaevyn
by • New Contributor III
  • 8744 Views
  • 4 replies
  • 0 kudos

Databricks SQL Warehouse did not auto stop after specified 90 minute interval - why not?

In this specific case, we're running a 2XSmall SQL Warehouse on Databricks SQL.In looking at the SQL Warehouse monitoring log for this cluster, we noticed:final query executed by user at 10:26 on 2023-06-20no activity for some time, yet cluster remai...

Data Engineering
Reply
  • 8744 Views
  • 4 replies
  • 0 kudos
View Replies
Latest Reply
msolcuadrado
New Contributor
  • 0 kudos

Hi! Was anyone able to solve this problem? I'm facing the same issue but without any queries being executed. It just magically turns on and never disables, and I don't see anything in the query history.

  • 0 kudos
3 More Replies

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels
Top Kudoed Authors
View all