cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Generative AI
Explore discussions on generative artificial intelligence techniques and applications within the Databricks Community. Share ideas, challenges, and breakthroughs in this cutting-edge field.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
Jayachithra
by • New Contributor
  • 146 Views
  • 1 replies
  • 1 kudos

Resolved! How to get MLflow OpenAI autolog traces from PySpark mapInPandas workers (and some pitfalls)

ContextI'm running an LLM pipeline on Databricks that distributes OpenAI API calls across Spark workers via mapInPandas. Getting mlflow.openai.autolog() to work on workers required solving three undocumented issues. Sharing here since I couldn't find...

Generative AI
Reply
  • 146 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
Louis_Frolio
Databricks Employee
  • 1 kudos

Greetings @Jayachithra , I did some digging and came up with some helpful tips/hints to help you along.   On Issue 1 (explicit MLflow context): expected behavior once you realize that mapInPandas spawns isolated Python worker processes, not threads. ...

  • 1 kudos
Labels