Databricks Community

We want to do demand forecasting for our supply chain. How should we benefit from Spark in the Usercase development?

We have a series of blogs on the topic which describe the challenges and the best practices on development of demand forecasting usecases on Databricks. Please refer to this blog and the references in it for more info.

How to register datasets for Detectron2

I'm trying to run a Detectron2 model in Databricks and cannot figure out how to register my train, val and test datasets. My datasets live in an Azure data lake. I have tried the following with no luck. Any help is appreciated. 1) Specifying full p...

i just can't seem to make a cluster without error

I have been trying to create a cluster and this is my first time using databricks. I have tried across multiple resources and am getting frustrated. Each time the cluster comes up with an error. I have no idea what I am doing wrong. I use default se...

pandas udf type grouped map fails

Hello, I am trying to get the shap values for my whole dataset using pandas udf for each category of a categorical variable. It runs well when I run it on a few categories but when I want to run the function on the whole dataset my job fails. I see ...

Can we schedule multiple Databricks jobs as pipeline with dependancies

Is it possible to turn off the redaction of secrets? Is there a better way to solve this?

As part of our Azure Data Factory pipeline, we utilize Databricks to run some scripts that identify which files we need to load from a certain source. This list of files is then passed back into Azure Data Factory utilizing the Exit status from the n...

Transactional approach to write to Azure ADLS gen2 storage

Hi, what is the recommended way to read data from delta table and write to ADLS gen2 storage in parquet format. In my case i use a notebook to read data do some processing and write it to storage and update delta table with detail of last written da...

How can I pass data between sql databases azure if the map is different?

I need to pass data between sql azure databases but the columns of some tables are different, the information must go to that column but with a different name.

For Databricks on AWS, when I set up a Kinesis Endpoint as shown here https://docs.databricks.com/administration-guide/cloud-configurations/aws/customer-managed-vpc.html#firewall-appliance-infrastructure should I use Kinesis Firehose or Kinesis Streams?

What is the differentiator between delta sharing and other cloud sharing platform

What is differentiator between delta sharing and other cloud sharing platform.

Logging Lifetime Plot_history_alive as a model

Hi Databricks Community, I have followed the CLV Databricks accelator (https://databricks.com/notebooks/CLV_Part_1_Customer_Lifetimes.html) to do an initial CLV analysis. Thank you for sharing this with the community. My question is that in the note...