Databricks Community

"dbfs:" quote added as a prefix to file path

There is a mount path /mnt/folder I am passing filename as a variable from another function and completing the path variable as follows: filename=file.txt path=/mnt/folder/subfolder/+filename When I'm trying to use the path variable is a function, f...

Kafka consumer and a web server simultaneously, thread blocking problem in microservice

assumptions: There are microservices behind an api-gateway, they communicate through HTTP synchronously. obviously, each one of those microservices is a web server. now I want my microservice to play as a Kafka producer and "consumer" too. more clea...

Double job execution caused by databricks' RemoteServiceExec using databricks-connector

Hello! I'm using databricks-connector to launch spark jobs using python. I've validated that the python version (3.8.10) and runtime version (8.1) are supported by the installed databricks-connect (8.1.10). Everytime a mapPartitions/foreachParti...

Bug Report: Date type with year less than 1000 (years 1-999) in spark sql where [solved]

Hi, I noticed unexpected behavior for Date type. If year value is less then 1000 then filtering do not work. Steps:create table test (date Date); insert into test values ('0001-01-01'); select * from test where date = '0001-01-01' Returns 0 rows....

How to create circles and find diameter in point cloud?

Good day, Everybody. I have a task I should create many circles in a point cloud (my file can be .csv) and calculate their diameter in the area 100m x 100 m. I have the coordinates of the starting point. The circles should be created at the height ...

We want to do demand forecasting for our supply chain. How should we benefit from Spark in the Usercase development?

We have a series of blogs on the topic which describe the challenges and the best practices on development of demand forecasting usecases on Databricks. Please refer to this blog and the references in it for more info.

How to register datasets for Detectron2

I'm trying to run a Detectron2 model in Databricks and cannot figure out how to register my train, val and test datasets. My datasets live in an Azure data lake. I have tried the following with no luck. Any help is appreciated. 1) Specifying full p...

i just can't seem to make a cluster without error

I have been trying to create a cluster and this is my first time using databricks. I have tried across multiple resources and am getting frustrated. Each time the cluster comes up with an error. I have no idea what I am doing wrong. I use default se...

pandas udf type grouped map fails

Hello, I am trying to get the shap values for my whole dataset using pandas udf for each category of a categorical variable. It runs well when I run it on a few categories but when I want to run the function on the whole dataset my job fails. I see ...

Can we schedule multiple Databricks jobs as pipeline with dependancies

Transactional approach to write to Azure ADLS gen2 storage

Hi, what is the recommended way to read data from delta table and write to ADLS gen2 storage in parquet format. In my case i use a notebook to read data do some processing and write it to storage and update delta table with detail of last written da...

Toggle titles to show by default for a user or notebook

I find titles to be useful in organizing my notebooks, but I don't like having to toggle the title display for each cell in order to add a title. Is there a way to toggle the UI to show titles by default for a user/notebook? This would be a good fea...