Databricks Community is an open-source platform for data enthusiasts and professionals to discuss, share insights, and collaborate on everything related to Databricks. Members can ask questions, share knowledge, and support each other in an environment that ensures respectful interactions.
Hi everyone, I'm building a Pyspark ML Pipeline where the first stage is to fill nulls with zero. I wrote a custom class to do this since I cannot find a Transformer that will do this imputation.&nb...
Hello Everyone. I want to explore LakeFlow Pipelines in the community version but don’t have access to Azure or AWS. I had a bad experience with Azure, where I was charged $85 while just trying ...
Hi Databricks support, I am looking for a standardized Databricks framework to update job definition using DevOps from non-production till it get productionized. Our current process of updating the D...
For a UK Government Agency, I made a Comprehensive presentation titled " Feature Engineering for Data Engineers: Building Blocks for ML Success". I made an article of it in Linkedlin toge...
We have tried to build a connection test logic to our software to try out the reachability of the SQL Warehouse, yet the connection parameters do not seem to function in expected manner. When the SQ...
Does anyone know how to do DevOps CICD pipeline to deploy changes to a Databricks Catalog? I've added a couple tables in DEV and need to deploy in Prod. Could'nt find any info on this anywhere.
We’re just a couple weeks removed from the biggest Data + AI Summit in history, where we introduced Databricks LakeFlow, a unified, intelligent solution for data engineering. While we are hard a...
Added support for customizing scatter point size and opacity in AI/BI dashboards. Improved interactivity in displaying tooltips when hovering over pie, scatter, and heatmap charts with many data poi...