cancel
Showing results for 
Search instead for 
Did you mean: 
Community Platform Discussions
Connect with fellow community members to discuss general topics related to the Databricks platform, industry trends, and best practices. Share experiences, ask questions, and foster collaboration within the community.
cancel
Showing results for 
Search instead for 
Did you mean: 

How to save the best model checkpoi through the epochs of a deep learning network through callbacks?

manupmanoos
New Contributor III

I have create a neural network and I am training the model with the code as below.  The code fails to write to the databricks file storage. is there any other way to write the checkpoint to databricks storage or to an s3 bucket directly?

custom_early_stopping = EarlyStopping(
    monitor='val_loss',
    patience=100,
)
best_model_checkpoint = ModelCheckpoint('local location in databricks file storge', save_best_only=True, monitor='val_loss', mode='min')
# Fit the model
history = model.fit(
           x = [inputs],
           y = y,
           batch_size= 10,
           epochs = 2,
           verbose=1,
           shuffle =False,
           validation_split=0.2,
           callbacks=[custom_early_stopping, best_model_checkpoint]
2 REPLIES 2

manupmanoos
New Contributor III

Hi @Retired_mod ,

I am not able to save it to local storage in databricks dbfs also. It is showing invalid operation when I am trying to save to databricks file storage. Additionally, I have valid aws credentials with which I am able to save a model to s3 buckets. What I am trying here is to save the best model checkpoint while training during the epochs to either the local or s3 storage

MeakhMalik
New Contributor II

I'm experiencing difficulty saving to nearby capacity in Data bricks DBFS due to an "invalid activity" message. In spite of the fact that I can save models to S3 with my AWS accreditations, I want to locally save the best model designated spot during preparing Spotify. For additional subtleties kindly take a look at sustenance world server. 

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group