cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

AbhishekNakka
by New Contributor II
  • 34 Views
  • 1 replies
  • 0 kudos

Databricks professional data engineer

Hi, i wanted to know i anyone has given databricks professional data engineering exam recently after oct 2025. I wanted to know if the syllabus has been updated or not ?

  • 34 Views
  • 1 replies
  • 0 kudos
Latest Reply
szymon_dybczak
Esteemed Contributor III
  • 0 kudos

Hi @AbhishekNakka ,Yes, the syllabus has been updated. The current exam objectives you can find at below link:Databricks Certified Data Engineer Professional September 2025 - Exam Guide.docxDatabricks Certified Data Engineer Professional | Databricks

  • 0 kudos
DatabricksEngi1
by New Contributor III
  • 95 Views
  • 4 replies
  • 0 kudos

Resolved! Problem in VS Code Extention

Until a few days ago, I was working with Databricks Connect using the VS Code extension, and everything worked perfectly.In my .databrickscfg file, I had authentication configured like this:  [name]host:token: When I ran my code, everything worked fi...

  • 95 Views
  • 4 replies
  • 0 kudos
Latest Reply
dkushari
Databricks Employee
  • 0 kudos

Hi @DatabricksEngi1 - Please ensure you have a Python Venv set up for each Python version that you use with Databricks Connect. Also, I have given step-by-step ways to debug the issue, clear the cache, etc [Read the files and instructions carefully b...

  • 0 kudos
3 More Replies
manugarri
by New Contributor II
  • 18686 Views
  • 12 replies
  • 2 kudos

Fuzzy text matching in Spark

I have a list of client provided data, a list of company names. I have to match those names with an internal database of company names. The client list can fit in memory (its about 10k elements) but the internal dataset is on hdfs and we use Spark ...

  • 18686 Views
  • 12 replies
  • 2 kudos
Latest Reply
Shamzaa3Q
New Contributor II
  • 2 kudos

+1 for rapidfuzz, I have used it in production pipelines. Better than just levenshtein function, as rapidfuzz provides a couple of other algorithms as well. I will warn you to not do what 2024 me attempted, which is use LLM to solve for this. It soun...

  • 2 kudos
11 More Replies
pranaav93
by New Contributor III
  • 53 Views
  • 1 replies
  • 0 kudos

Resolved! TransformWithState is not emitting for live streams

Hi Team, For one of my custom logics i went with transformwithState processor. However it is not working for live stream inputs., I have a start date filter on my df_base and when I give start date that is not current, the processor computes df_loss ...

Data Engineering
apachespark
pyspark
StatefulStreaming
StructuredStreaming
transformWithState
  • 53 Views
  • 1 replies
  • 0 kudos
Latest Reply
pranaav93
New Contributor III
  • 0 kudos

I managed to solve this. The issue was with how I handled the value state in the def init method. It was handled as a dataframe which caused the state to never materialize nor update therefore emitting nulls.I changed them to a tuple of values and th...

  • 0 kudos
drag7ter
by Contributor
  • 2398 Views
  • 1 replies
  • 0 kudos

Delta sharing view and cached data in DSFF

I've created a view with row level access based on CURRENT_RECIPIENT() function in the where clause. And I have 100s of clients as recipients that query this view.The problem is, when I modify this view CREATE OR REPLACE with a new sql code, and reci...

  • 2398 Views
  • 1 replies
  • 0 kudos
Latest Reply
AbhaySingh
New Contributor
  • 0 kudos

Have you tried something like this already?  Force Cache Invalidation (Recommended)  -- After CREATE OR REPLACE VIEW, execute:  ALTER SHARE <share_name> REMOVE TABLE <schema>.<view_name>;  ALTER SHARE <share_name> ADD TABLE <schema>.<view_name>;  Thi...

  • 0 kudos
abhijit007
by New Contributor
  • 1944 Views
  • 1 replies
  • 0 kudos

Lakebridge code conversion | Permission issue

Hi,I’ve successfully installed the transpile module from Lakebridge and tried the tool to convert Informatica mappings into PySpark code. However, I’m encountering a PermissionError during execution. I’ve provided the relevant environment details and...

Data Engineering
Lakebridge
Warehouse Migration
  • 1944 Views
  • 1 replies
  • 0 kudos
Latest Reply
dkushari
Databricks Employee
  • 0 kudos

Hi @abhijit007 - I see that this has been resolved in the 0.10.5 release. Can you please retest and confirm?

  • 0 kudos
AlexSantiago
by New Contributor II
  • 6813 Views
  • 20 replies
  • 4 kudos

spotify API get token - raw_input was called, but this frontend does not support input requests.

hello everyone, I'm trying use spotify's api to analyse my music data, but i'm receiving a error during authentication, specifically when I try get the token, above my code.Is it a databricks bug?pip install spotipyfrom spotipy.oauth2 import SpotifyO...

  • 6813 Views
  • 20 replies
  • 4 kudos
Latest Reply
Alyceveum25
  • 4 kudos

Thank you 

  • 4 kudos
19 More Replies
raghvendrarm1
by New Contributor
  • 146 Views
  • 2 replies
  • 1 kudos

Results from the spark application to driver

I tried to read many articles but still not clear on this:The executors complete the execution of tasks and have the results with them.1. The results(output data) from all executors is transported to driver in all cases or executors persist it if tha...

  • 146 Views
  • 2 replies
  • 1 kudos
Latest Reply
K_Anudeep
Databricks Employee
  • 1 kudos

Hello @raghvendrarm1  ,   Below are the answers to your questions: Do executors always send “results” to the driver? No. Only actions that return values (e.g., collect, take, first, count) bring data back to the driver. collect explicitly “returns al...

  • 1 kudos
1 More Replies
Saf4Databricks
by New Contributor III
  • 185 Views
  • 3 replies
  • 0 kudos

Cannot import pyspark.pipelines module

Question: What could be a cause of the following error of my code in a Databricks notebook, and how can we fix the error? I'm using latest Free Edition of Databricks that has runtime version 17.2 and PySpark version 4.0.0.Error:ImportError: cannot im...

  • 185 Views
  • 3 replies
  • 0 kudos
Latest Reply
dkushari
Databricks Employee
  • 0 kudos

Hi @Saf4Databricks - Are you trying to use it from a standalone Databricks notebook? You should only use it from with Lakeflow Declarative Pipeline (LDP). The link you shared is about LDP. Here is an example where I used it.    

  • 0 kudos
2 More Replies
TalessRocha
by New Contributor II
  • 1321 Views
  • 10 replies
  • 8 kudos

Resolved! Connect to azure data lake storage using databricks free edition

Hello guys, i'm using databricks free edition (serverless) and i am trying to connect to a azure data lake storage.The problem I'm having is that in the free edition we can't configure the cluster so I tried to make the connection via notebook using ...

  • 1321 Views
  • 10 replies
  • 8 kudos
Latest Reply
BS_THE_ANALYST
Esteemed Contributor II
  • 8 kudos

@TalessRocha thanks for getting back to us! Glad to hear you got it working, that's awesome. Best of luck with your projects.All the best,BS

  • 8 kudos
9 More Replies
maninegi05
by New Contributor
  • 171 Views
  • 2 replies
  • 0 kudos

DLT Pipeline Stopped working

Hello, Suddenly our DLT pipelines we're getting failures saying thatLookupError: Traceback (most recent call last): result_df = result_df.withColumn("input_file_path", col("_metadata.file_path")).withColumn( ...

  • 171 Views
  • 2 replies
  • 0 kudos
Latest Reply
Khaja_Zaffer
Contributor III
  • 0 kudos

 May be there is internally some updates from databricks Can Check and Switch Your Pipeline Channel, In the DLT pipeline settings (under Advanced > Channel), confirm if it's set to "Preview". Switch to "Current" for a more stable engine version, then...

  • 0 kudos
1 More Replies
Malthe
by Contributor II
  • 266 Views
  • 4 replies
  • 0 kudos

Resolved! Can't enable "variantType-preview" using DLTs

Using create_streaming_table and passing table properties as follows, I get an error running the pipeline for the first time:> Your table schema requires manually enablement of the following table feature(s): variantType-preview.I'm using this code:c...

  • 266 Views
  • 4 replies
  • 0 kudos
Latest Reply
Malthe
Contributor II
  • 0 kudos

There's a workaround available in most situations which is to first create the table without the VARIANT column, run the pipeline at least once, and then add the column in a subsequent refresh.

  • 0 kudos
3 More Replies
Upendra_Dwivedi
by Contributor
  • 2496 Views
  • 1 replies
  • 1 kudos

Databricks APP OBO User Authorization

Hi All,We are using on-behalf of user authorization method for our app and the x-forwarded-access-token is expiring after sometime and we have to redeploy our app to rectify the issue. I am not sure what is the issue or how we can keep the token aliv...

Upendra_Dwivedi_0-1747911721728.png
  • 2496 Views
  • 1 replies
  • 1 kudos
Latest Reply
jamesl
Databricks Employee
  • 1 kudos

Hi @Upendra_Dwivedi , are you still facing this issue? The x-forwarded-access-token your app receives is the current user’s access token that Databricks forwards in HTTP headers for on‑behalf‑of‑user access. You should read it from the request on eac...

  • 1 kudos

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local community—sign up today to get started!

Sign Up Now
Labels