cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

spark.sparkContext.setCheckpointDir - External Azure Storage

Go to solution
sintsan
New Contributor II

‎04-25-2023 01:40 AM

Is it possible to direct spark.sparkContext.setCheckpointDir to an external Azure Storage Container location (instead of DBFS) & if so how, there's very little documentation on that.

0 Kudos
1 ACCEPTED SOLUTION

Accepted Solutions

Go to solution
-werners-
Esteemed Contributor III

‎04-25-2023 07:32 AM

yes,

the directory must be an HDFS path if running on a cluster.

All you need to do is provide the correct path.

View solution in original post

1 REPLY 1

Go to solution
-werners-
Esteemed Contributor III

‎04-25-2023 07:32 AM

yes,

the directory must be an HDFS path if running on a cluster.

All you need to do is provide the correct path.

Announcements
Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.