topic Re: [Databricks User Research] AI Assistance for Data Science Work in Machine Learning

[Databricks User Research] AI Assistance for Data Science Work

DataBrickator — Tue, 26 Aug 2025 22:32:16 GMT

Hi all! I'm running a user research series on the topic of AI Assistance for data science work (e.g. exploratory data analysis, feature engineering, model eval, etc.). If you are a Databricks Notebooks user, I would love to pick your brain on your preferences for AI Assistance via a 60 minute conversation sometime between 08/27 to 09/05. If this sounds like you, fill out this form!

For the selected research participants, as a thank you gift for your time we are able to offer a $150 Visa card, as long as your company policy allows for it.

Hope to connect with you soon!

Re: [Databricks User Research] AI Assistance for Data Science Work

Advika — Thu, 28 Aug 2025 10:22:37 GMT

Thanks for sharing, @DataBrickator! This is an amazing chance for our Databricks Notebook users to share feedback directly with the team.

Re: [Databricks User Research] AI Assistance for Data Science Work

nayan_wylde — Thu, 28 Aug 2025 15:39:32 GMT

I would like to participate. I have been a Databricks Assistant user for long.

Re: [Databricks User Research] AI Assistance for Data Science Work

ck7007 — Thu, 28 Aug 2025 16:15:26 GMT

@DataBrickatorPerfect timing! I'm a heavy Databricks Notebooks user working on performance optimization and ML pipelines.

My daily workflows involve:

EDA on 100M+ record Iceberg tables
Real-time feature engineering with Spark Streaming
Custom indexing strategies (zonemap + Bloom filters)

Key AI assistance challenges I face:

AI tools suggesting pandas instead of Spark operations
Lack of awareness about distributed computing costs
Generic suggestions that ignore Databricks-specific optimizations

Currently documenting how AI assistance impacts my productivity—have concrete examples where it saves 70% of time but requires manual optimization for scale.

Would love to discuss how AI could better understand Spark's execution model and suggest performance optimizations.

Filling out the form now—excited to contribute to making AI assistance more effective for data scientists!

Question: Will Unity Catalog integration be part of the discussion?

Re: [Databricks User Research] AI Assistance for Data Science Work

DataBrickator — Thu, 28 Aug 2025 20:00:48 GMT

Thanks for sharing your use of Notebooks!
To answer your question regarding Unity Catalog, we can touch on it and how it relates to your Notebook use, but the research mostly will focus on AI Assistance within the Notebook itself. I'll reach out once I get your form submission.