Databricks Community

pandas udf type grouped map fails

Hello, I am trying to get the shap values for my whole dataset using pandas udf for each category of a categorical variable. It runs well when I run it on a few categories but when I want to run the function on the whole dataset my job fails. I see ...

Can we schedule multiple Databricks jobs as pipeline with dependancies

Is it possible to turn off the redaction of secrets? Is there a better way to solve this?

As part of our Azure Data Factory pipeline, we utilize Databricks to run some scripts that identify which files we need to load from a certain source. This list of files is then passed back into Azure Data Factory utilizing the Exit status from the n...

Transactional approach to write to Azure ADLS gen2 storage

Hi, what is the recommended way to read data from delta table and write to ADLS gen2 storage in parquet format. In my case i use a notebook to read data do some processing and write it to storage and update delta table with detail of last written da...

Toggle titles to show by default for a user or notebook

I find titles to be useful in organizing my notebooks, but I don't like having to toggle the title display for each cell in order to add a title. Is there a way to toggle the UI to show titles by default for a user/notebook? This would be a good fea...

How can I pass data between sql databases azure if the map is different?

I need to pass data between sql azure databases but the columns of some tables are different, the information must go to that column but with a different name.

Accessing databricks from Presto SSQL

What's the best way to federate a query to delta lake or the databricks from presto sql without having to create external tables? PrestoSQL doesn't have access to S3. Can PrestoSQL be configured with jdbc driver or plugin?

For Databricks on AWS, when I set up a Kinesis Endpoint as shown here https://docs.databricks.com/administration-guide/cloud-configurations/aws/customer-managed-vpc.html#firewall-appliance-infrastructure should I use Kinesis Firehose or Kinesis Streams?

What is the differentiator between delta sharing and other cloud sharing platform

What is differentiator between delta sharing and other cloud sharing platform.

Logging Lifetime Plot_history_alive as a model

Hi Databricks Community, I have followed the CLV Databricks accelator (https://databricks.com/notebooks/CLV_Part_1_Customer_Lifetimes.html) to do an initial CLV analysis. Thank you for sharing this with the community. My question is that in the note...

Databricks Error in SQL statement: ParseException: mismatched input 'Service_Date

I am running this script in Azure Databricks using spark SQL , getting this error below , I am new to Databricks so wondering any tips on troubleshooting thisError in SQL statement: ParseException:mismatched input 'Service_Date' expecting {'(', 'DESC...

Read metadata through JDBC driver

Dear all, The Spark JDBC driver (SparkJDBC42.jar) is unable to capture certain information from the below table structure: 1. table level comment 2. the TBLPROPERTIES key-value pair information 3. PARTITION BY information However, it captures the co...

Incremental updates to s3 csv files, autoloader, and delta lake updates

I'm using the Databricks autoloader to incrementally load a series of csv files on s3 which I update with an API. My tyipcal work process is to update only the latest year file each night. But, there are ocassions where previous years also get update...