topic Re: How can I save a keras model from a python notebook in databricks to an s3 bucket? in Machine Learning

How can I save a keras model from a python notebook in databricks to an s3 bucket?

manupmanoos — Wed, 09 Aug 2023 12:51:50 GMT

I have a trained model on Databricks python notebook. How can I save this to an s3 bucket.

Re: How can I save a keras model from a python notebook in databricks to an s3 bucket?

Kumaran — Mon, 14 Aug 2023 06:48:58 GMT

Hi @manupmanoos,

Thank you for posting your question in Databricks community.

Here are the steps to save a Keras model from a Python notebook in Databricks to AWS S3 bucket:

Install the AWS SDK and set up your credentials using Databricks Secret Manager or environment variables.
Train a Keras model, for example using model = keras.models.Sequential().
Save the model locally using model.save('/dbfs/models/model.h5') command.
Use the aws command to copy the saved model file to an S3 bucket. Here is an example:

# Set credentials and create S3 client import boto3 import os aws_access_key_id = dbutils.secrets.get(scope="<scope-name>", key="<key-name>") aws_secret_access_key = dbutils.secrets.get(scope="<scope-name>", key="<key-name>") os.environ['AWS_ACCESS_KEY_ID'] = aws_access_key_id os.environ['AWS_SECRET_ACCESS_KEY'] = aws_secret_access_key s3_client = boto3.client('s3') # Upload the saved model from local file system to S3 bucket s3_bucket = "<bucket-name>" s3_prefix = "<bucket-prefix>" model_path = "/dbfs/models/model.h5" s3_key = "{}/model.h5".format(s3_prefix) s3_client.upload_file(model_path, s3_bucket, s3_key)

Re: How can I save a keras model from a python notebook in databricks to an s3 bucket?

manupmanoos — Tue, 15 Aug 2023 15:02:58 GMT

Hi @Kumaran ,

Could you please let me know how we would we load the same saved model back to a databricks notebook?

Thanks,

Manu

Re: How can I save a keras model from a python notebook in databricks to an s3 bucket?

Kumaran — Tue, 15 Aug 2023 15:38:14 GMT

Hi @manupmanoos,

Thank you for posting your question in the Databricks community.

Below is an example of how to load the same saved model back to a Databricks notebook

from keras.models import load_model

model = load_model('/dbfs/models/model.h5')

Re: How can I save a keras model from a python notebook in databricks to an s3 bucket?

manupmanoos — Tue, 15 Aug 2023 15:44:51 GMT

Hi @Kumaran ,

Sorry, I was not clear with my question. How do we load the saved model back from the s3 bucket?

Thanks,

Manu

Re: How can I save a keras model from a python notebook in databricks to an s3 bucket?

Kumaran — Tue, 15 Aug 2023 17:35:46 GMT

Hi @manupmanoos,

Please check the below code on how to load the saved model back from the s3 bucket

import boto3 import os from keras.models import load_model # Set credentials and create S3 client aws_access_key_id = dbutils.secrets.get(scope="<scope-name>", key="<key-name>") aws_secret_access_key = dbutils.secrets.get(scope="<scope-name>", key="<key-name>") os.environ['AWS_ACCESS_KEY_ID'] = aws_access_key_id os.environ['AWS_SECRET_ACCESS_KEY'] = aws_secret_access_key s3_client = boto3.client('s3') # Specify the S3 bucket and model file path s3_bucket = "<bucket-name>" s3_prefix = "<bucket-prefix>" s3_key = "{}/model.h5".format(s3_prefix) # Download the model file from S3 local_model_path = "/dbfs/models/model.h5" s3_client.download_file(s3_bucket, s3_key, local_model_path) # Load the model using Keras loaded_model = load_model(local_model_path)