[Databricks User Research] AI Assistance for Data Science Work

[DataBrickator]

Hi all! I'm running a user research series on the topic of AI Assistance for data science work (e.g. exploratory data analysis, feature engineering, model eval, etc.). If you are a Databricks Notebooks user, I would love to pick your brain on your preferences for AI Assistance via a 60 minute conversation sometime between 08/27 to 09/05. If this sounds like you, fill out this form!

For the selected research participants, as a thank you gift for your time we are able to offer a $150 Visa card, as long as your company policy allows for it.

Hope to connect with you soon!

[Advika]

Thanks for sharing, @DataBrickator! This is an amazing chance for our Databricks Notebook users to share feedback directly with the team.

[nayan_wylde]

I would like to participate. I have been a Databricks Assistant user for long.

[ck7007]

@DataBrickatorPerfect timing! I'm a heavy Databricks Notebooks user working on performance optimization and ML pipelines.

My daily workflows involve:

• EDA on 100M+ record Iceberg tables
• Real-time feature engineering with Spark Streaming
• Custom indexing strategies (zonemap + Bloom filters)

Key AI assistance challenges I face:

• AI tools suggesting pandas instead of Spark operations
• Lack of awareness about distributed computing costs
• Generic suggestions that ignore Databricks-specific optimizations

Currently documenting how AI assistance impacts my productivity—have concrete examples where it saves 70% of time but requires manual optimization for scale.

Would love to discuss how AI could better understand Spark's execution model and suggest performance optimizations.

Filling out the form now—excited to contribute to making AI assistance more effective for data scientists!

Question: Will Unity Catalog integration be part of the discussion?

[DataBrickator · Accepted solution]

Thanks for sharing your use of Notebooks! 
To answer your question regarding Unity Catalog, we can touch on it and how it relates to your Notebook use, but the research mostly will focus on AI Assistance within the Notebook itself. I'll reach out once I get your form submission.