- 5577 Views
- 8 replies
- 9 kudos
Resolved! Pandas.spark.checkpoint() doesn't broke lineage
Hi, I'm doing some something simple on Databricks notebook:spark.sparkContext.setCheckpointDir("/tmp/") import pyspark.pandas as ps sql=("""select field1, field2 From table Where date>='2021-01.01""") df = ps.sql(sql) df.spark.checkpoint()That...
- 5577 Views
- 8 replies
- 9 kudos
- 9 kudos
checkpoint() returns a checkpointed DataFrame, so you need to assign it to a new variable:checkpointedDF = df.checkpoint()
- 9 kudos
- 5091 Views
- 8 replies
- 3 kudos
Resolved! Spark Streaming - Checkpoint State EOF Exception
I have a Spark Structured Streaming job which reads from 2 Delta tables in streams , processes the data and then writes to a 3rd Delta table. The job is being run with the Databricks service on GCP.Sometimes the job fails with the following exception...
- 5091 Views
- 8 replies
- 3 kudos
- 3 kudos
Hi @Jose Gonzalez​ ,Do you require any more information regarding the code? Any idea what could be cause for the issue?Thanks and Regards,Rohan
- 3 kudos