Databricks Community

Introduction Data and AI media will try to convince you of two terribly misguided ideas about AI’s impact on analytics. First of all, there’s the idea that prompt engineers who know just the right phrase to unlock the secret powers of generative AI w...

Introduction MLflow makes it easy to log and deploy machine learning models for use across your organization. However, the standard MLflow interfaces may not provide the necessary functionality to achieve every use case. For example, a simple scikit-...

Introduction If you’re coming from a Pandas background, moving from the simple Pandas on Spark API into the more flexible Pandas function paradigms can be very intimidating. More importantly, it’s very confusing to distinguish between what appears to...

  Introduction Databricks and DSPy can help overcome common challenges of creating Compound AI systems, including ones tasked with writing blog articles like the one you are reading right now. The examples in this article use Databricks Foundation ...

In this short tutorial, we’ll implement an approach to making certain applyInPandas operations run many times faster. First, let's generate some dummy data for this example using Spark. For our example, we’ll create a function that returns a datafram...