cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Machine Learning
Dive into the world of machine learning on the Databricks platform. Explore discussions on algorithms, model training, deployment, and more. Connect with ML enthusiasts and experts.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

What added advantage would using an Sql Alchemy database have, while I have an artifact destination for my MlFlow server?

naveen_marthala
Contributor

I am setting up mlflow server with Postgres and S3 on AWS ECS(or AWS EC2) for personal usage. I would like to know if using Postgres would actually give me any benefit?

as shown in scenario 5 in docs, I would like to set up server with proxied artifact storage, but without postgres in AWS RDS and also not in `--artifacts-only` and `--artifacts-only` modes as shown in scenario 6 in docs. So, if i launch a mlflow tracking server like this, will all the metadata that usually gets saved to an SQLAlchemy database(postgres in my case) be saved to the location of attribute `--artifacts-destination` as files?

I would like to do this to save costs from AWS RDS that runs Postgres.

If you aren't familiar with AWS ECS, it is a fully managed service (a simpler version of kubernetes) that runs container base applications. and EC2 is just user managed machines or servers.

2 REPLIES 2

Kaniz_Fatma
Community Manager
Community Manager

Hi @Naveen Marthalaโ€‹, Here is a step-by-step guide to set up MLflow with a Postgres DB for storing metadata and a systemd unit to keep it running. Please have a look and let us know if that helps.

Thanks for trying to help me @Kaniz Fatmaโ€‹ .

My question was not to learn how to set up mlflow with postgres. But, to learn what additional advantage would using an sql-alchemy database like postgres would have, instead of storing all the metadata to files. that is, what advantage would I have, if I use an sql-alchemy for backend and not use one. please note that, the mlflow server will not be in '--artifacts-only' mode?

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group