cancel
Showing results for 
Search instead for 
Did you mean: 
Machine Learning
cancel
Showing results for 
Search instead for 
Did you mean: 

What added advantage would using an Sql Alchemy database have, while I have an artifact destination for my MlFlow server?

naveen_marthala
Contributor

I am setting up mlflow server with Postgres and S3 on AWS ECS(or AWS EC2) for personal usage. I would like to know if using Postgres would actually give me any benefit?

as shown in scenario 5 in docs, I would like to set up server with proxied artifact storage, but without postgres in AWS RDS and also not in `--artifacts-only` and `--artifacts-only` modes as shown in scenario 6 in docs. So, if i launch a mlflow tracking server like this, will all the metadata that usually gets saved to an SQLAlchemy database(postgres in my case) be saved to the location of attribute `--artifacts-destination` as files?

I would like to do this to save costs from AWS RDS that runs Postgres.

If you aren't familiar with AWS ECS, it is a fully managed service (a simpler version of kubernetes) that runs container base applications. and EC2 is just user managed machines or servers.

2 REPLIES 2

Kaniz
Community Manager
Community Manager

Hi @Naveen Marthala​, Here is a step-by-step guide to set up MLflow with a Postgres DB for storing metadata and a systemd unit to keep it running. Please have a look and let us know if that helps.

Thanks for trying to help me @Kaniz Fatma​ .

My question was not to learn how to set up mlflow with postgres. But, to learn what additional advantage would using an sql-alchemy database like postgres would have, instead of storing all the metadata to files. that is, what advantage would I have, if I use an sql-alchemy for backend and not use one. please note that, the mlflow server will not be in '--artifacts-only' mode?

Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.