Pyspark Merge parquet and delta file

alesventus — Tue, 20 Jun 2023 09:02:48 GMT

Is it possible to use merge command when source file is parquet and destination file is delta? Or both files must delta files?

Currently, I'm using this code and I transform parquet into delta and it works. But I want to avoid of this tranformation.

Thanks

from delta.tables import *
 
deltaTablePeople = DeltaTable.forPath(spark, 'abfss://destination-delta')
deltaTablePeopleUpdates = DeltaTable.forPath(spark, 'abfss://source-parquet')
 
dfUpdates = deltaTablePeopleUpdates.toDF()
 
deltaTablePeople.alias('people') \
  .merge(
    dfUpdates.alias('updates'),
    'people.id = updates.id'
  ) \
  .whenMatchedUpdate(set =...

Re: Pyspark Merge parquet and delta file

Anonymous — Wed, 21 Jun 2023 03:16:32 GMT

Hi @Ales ventus

We haven't heard from you since the last response from @Kaniz Fatma , and I was checking back to see if her suggestions helped you.

Or else, If you have any solution, please share it with the community, as it can be helpful to others.

Also, Please don't forget to click on the "Select As Best" button whenever the information provided helps resolve your question.

topic Pyspark Merge parquet and delta file in Machine Learning

Pyspark Merge parquet and delta file

Re: Pyspark Merge parquet and delta file