Hi everyone,   I recently worked on designing a scalable machine learning evaluation pipeline using Databricks. The goal was to automate model scoring and analysis across large datasets while maintaining version control, metric tracking, and reproducibility.   Databricks’ distributed compute environment, along with its strong support for orchestration and experimentation, helped us streamline our end-to-end evaluation workflow. Key focus areas included:   Parallelizing model inference for high throughput Tracking evaluation metrics and runs consistently Managing dataset and result versioning with ease     If you’re exploring similar use cases or have insights on optimizing ML evaluation frameworks in Databricks, I’d love to hear from you. Open to sharing technical details and learning from others in the community. ... View more

Join the Bay Area Databricks Community for a data meetup where innovation meets hands-on insights. Connect with data leaders, explore real-world use cases, and discover how modern analytics and AI are transforming industries. Whether you're a data engineer, analyst, or business leader, this is your chance to network, share ideas, and gain actionable strategies to drive smarter decisions. Agenda - Introductions: Sigma & West Monroe - Presentation: Powering Autonomy with Data: Rivian’s Scalable ADAS Analytics Framework Learn how Rivian uses Databricks to scale their data analytics and drive powerful insights. Don’t miss this opportunity to connect, learn, and grow with the Bay Area Databricks Community! 👉 RSVP now to secure your spot! https://events.sigmacomputing.com/databricksmeetupsigma Hosted by Sigma Computing and West Monroe ... View more

The Databricks community team needs your help to lead and shape this local community. Why You Should Step Up: Membership: You are already a member of the Databricks community in this region, which gives you a unique perspective on topics relevant to this group Leadership: Build leadership skills by guiding the group Visibility: Stand out in the Databricks community and unlock new leaderboard levels Impact: Help to shape the future of Databricks in your region by leading discussion topics and events Your Role: Organize/host in-person and virtual meetup events, in conjunction with the Databricks community team Keep the group active between events by starting discussions, sharing insights, and asking questions Welcome new members Interested? Comment below or email community@databricks.com. The Databricks community team is here to provide whatever support you need. Thanks Rishabh ... View more

Hi Bay Area Databricks Community members!  I am happy to announce that while we are in the early stages of planning our next event, we are targeting Thursday, April 24 from 5 - 7:30 PM.  We will have more details available in the upcoming weeks leading up to the event but please save the date! ... View more

  📅 Thursday, March 6, 2025 🕗 8:00 AM – 5:00 PM PT 📍 The Ameswell Hotel, 800 Moffett Blvd, Mountain View, CA 94043 🔗 Secure your spot today! Register Here! Join us for a half-day deep dive into the future of data and AI. Learn how to: Eliminate data silos and simplify governance Empower teams with AI-driven insights using natural language Develop high-quality AI applications while maintaining data privacy Streamline data management across engineering, governance, and analytics What’s in store? Keynote & Customer Success Stories Expert-led Technical Sessions Networking Opportunities Agenda Highlights: 8:00 AM – 9:00 AM | Breakfast & Registration 9:00 AM – 9:25 AM | Welcome + Keynote 9:25 AM – 9:40 AM | Customer Story 9:40 AM – 10:10 AM | Simplify Data and AI Governance at Scale With Unity Catalog 10:10 AM – 10:35 AM | Break & Networking 10:35 AM – 11:05 AM | Build and Deploy AI Apps With Databricks 11:05 AM – 11:35 AM | AI-Powered Data Warehousing With Databricks SQL 11:35 AM – 11:50 AM | Partner Session 11:50 AM – 12:00 PM | Closing Remarks 12:00 PM – 1:00 PM | Lunch & Networking   ... View more