Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-13-2023 11:47 PM - edited 08-13-2023 11:48 PM
Hi @manupmanoos,
Thank you for posting your question in Databricks community.
Here are the steps to save a Keras model from a Python notebook in Databricks to AWS S3 bucket:
- Install the AWS SDK and set up your credentials using Databricks Secret Manager or environment variables.
- Train a Keras model, for example using model = keras.models.Sequential().
- Save the model locally using model.save('/dbfs/models/model.h5') command.
- Use the aws command to copy the saved model file to an S3 bucket. Here is an example:
# Set credentials and create S3 client
import boto3
import os
aws_access_key_id = dbutils.secrets.get(scope="<scope-name>", key="<key-name>")
aws_secret_access_key = dbutils.secrets.get(scope="<scope-name>", key="<key-name>")
os.environ['AWS_ACCESS_KEY_ID'] = aws_access_key_id
os.environ['AWS_SECRET_ACCESS_KEY'] = aws_secret_access_key
s3_client = boto3.client('s3')
# Upload the saved model from local file system to S3 bucket
s3_bucket = "<bucket-name>"
s3_prefix = "<bucket-prefix>"
model_path = "/dbfs/models/model.h5"
s3_key = "{}/model.h5".format(s3_prefix)
s3_client.upload_file(model_path, s3_bucket, s3_key)