Kumaran
Databricks Employee
Databricks Employee

Hi @manupmanoos,

Thank you for posting your question in Databricks community.

Here are the steps to save a Keras model from a Python notebook in Databricks to AWS S3 bucket:

  1. Install the AWS SDK and set up your credentials using Databricks Secret Manager or environment variables.
  2. Train a Keras model, for example using model = keras.models.Sequential().
  3. Save the model locally using model.save('/dbfs/models/model.h5') command.
  4. Use the aws command to copy the saved model file to an S3 bucket. Here is an example:

 

# Set credentials and create S3 client
import boto3
import os

aws_access_key_id = dbutils.secrets.get(scope="<scope-name>", key="<key-name>")
aws_secret_access_key = dbutils.secrets.get(scope="<scope-name>", key="<key-name>")
os.environ['AWS_ACCESS_KEY_ID'] = aws_access_key_id
os.environ['AWS_SECRET_ACCESS_KEY'] = aws_secret_access_key

s3_client = boto3.client('s3')

# Upload the saved model from local file system to S3 bucket
s3_bucket = "<bucket-name>"
s3_prefix = "<bucket-prefix>"

model_path = "/dbfs/models/model.h5"
s3_key = "{}/model.h5".format(s3_prefix)

s3_client.upload_file(model_path, s3_bucket, s3_key)

 

 

View solution in original post