cancel
Showing results for 
Search instead for 
Did you mean: 
Machine Learning
Dive into the world of machine learning on the Databricks platform. Explore discussions on algorithms, model training, deployment, and more. Connect with ML enthusiasts and experts.
cancel
Showing results for 
Search instead for 
Did you mean: 

Pyspark Merge parquet and delta file

alesventus
Contributor

Is it possible to use merge command when source file is parquet and destination file is delta? Or both files must delta files?

Currently, I'm using this code and I transform parquet into delta and it works. But I want to avoid of this tranformation.

Thanks

from delta.tables import *
 
deltaTablePeople = DeltaTable.forPath(spark, 'abfss://destination-delta')
deltaTablePeopleUpdates = DeltaTable.forPath(spark, 'abfss://source-parquet')
 
dfUpdates = deltaTablePeopleUpdates.toDF()
 
deltaTablePeople.alias('people') \
  .merge(
    dfUpdates.alias('updates'),
    'people.id = updates.id'
  ) \
  .whenMatchedUpdate(set =...

1 REPLY 1

Anonymous
Not applicable

Hi @Ales ventus​ 

We haven't heard from you since the last response from @Kaniz Fatma​ , and I was checking back to see if her suggestions helped you.

Or else, If you have any solution, please share it with the community, as it can be helpful to others. 

Also, Please don't forget to click on the "Select As Best" button whenever the information provided helps resolve your question.

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group