Hubert-Dudek
Databricks MVP

at least in Azure MLFlow limits are quite strict per workspace:

  • Low throughput experiment management (list, update, delete, restore): 7 qps
  • Search runs: 7 qps
  • Log batch: 47 qps
  • All other APIs: 127 qps

qps - queries per second. In addition, there is a limit of 20 concurrent model versions in Pending status (in creation) per workspace. Additionally 429 are automatically retry.

Is models trained in parallel for every group? Maybe instead of parallel just train one group by one and monitor executor usage as anyway it can be close to 100% and can take the same time.


My blog: https://databrickster.medium.com/

View solution in original post