Databricks Community

Serverless Model Serving does not mount the UC Volumes FUSE path (/Volumes), so references to “/Volumes/…” inside a custom pyfunc’s model code will fail at container build or runtime. The correct pattern is to package any required files (like your ...

Sorry, I just found out while checking the official documentation that starting from November, dependency JAR files are supported.I’ll give it a try and see how well it works.If anyone has any insights, I would greatly appreciate your guidance.

Hi @kevin11 ,I guess it's their standard way of library deprecation policy. In their docs they mentioned that when a library is planned for removal, Databricks takes following steps to notify customers:So they've added those note to AutoMl docs:And y...

You can define specific resources by target in DAB as shown here. This is valid for jobs and/or tasks:For instance, in my case:I think, best option (but not available as far as I know) would be to be able to define "include" sections by target, inste...

Distributed training with PyTorch’s DistributedDataParallel (DDP) is not inherently fault-tolerant—if any node fails, the whole job crashes, and, as you noted, checkpointing cannot auto-recover the process without hypervisor or application-level orch...

You are noticing a common pattern in Databricks ML tutorials: data is often processed with Spark for scalability, but training and modeling are frequently done on pandas DataFrames using single-node libraries like scikit-learn. This workflow can be c...

When dealing with feature tables and null values—especially via Databricks Feature Engineering objects (but also more broadly in Spark or feature platforms)—there are some nuanced behaviors when schema inference is required. Here are clear answers to...

@ryojikn and @irtizak , you’re right. Databricks Model Serving allows splitting traffic between model versions, but it doesn’t have a true shadow deployment where live production traffic is mirrored to a new model for monitoring without affecting use...

Developing ETL pipelines in Databricks comes with challenges like managing diverse data sources, optimizing Spark performance, and controlling cloud costs. Ensuring data quality, handling errors, and maintaining security and compliance add complexity...