Databricks Community

@pjv Can you please try the following, you'll basically want to have more than a single partition: from pyspark.sql import SparkSession from pyspark.sql.functions import udf from pyspark.sql.types import StringType # Initialize Spark session (if not...

@Vasu_Kumar_T can you provide some more details or context? Feel free to replace sensitive data. Where are you getting this? How are you passing the keys to decrypt a file? Is there a move comprehensive stacktrace apart from this message in the image...

Thanks for reporting this Sangram. Are these youtube and educational contents in the Databricks channel? > set delta.databricks.delta.retentionDurationCheck.enabled = false. It works if I want to delete obsolete files whose lifespan is less than defa...

Hey, This is by design. I understand the jobs are failing when run on a UC Single user cluster since it is unable to install a Jar package located in the /Workspace path. This is however a known behaviour and is already documented below: https://docs...

https://kb.databricks.com/unity-catalog/invalid_parameter_valuelocation_overlap-overlaps-with-managed-storage-error 

Okay, so I checked some documents - The email addresses will also be case-insensitive, the same behavior as in AWS, Azure and GCP. This means that email addresses will be stored in lowercase in Databricks. So, the issue is not with case sensitivity b...

Hi @ElaPG1 , While the cluster sounds like a pretty good one with Autoscaling, it depends on the workload too. The Standard_D8s_v5 instances you are using have 32GB memory and 8 cores. While these are generally good, you might want to experiment with...

Hi @Garrus990 , To pass a parameter to a task that is based on a UNIX function, you can use the Databricks Jobs API to dynamically calculate dates with respect to the date of running your job.  Use a Notebook to Calculate Dates: Create a notebook tha...

Hi @mlopsuser , For organizing Databricks Asset Bundles (DABs) in your scenario with two separate regression models and datasets, it is generally recommended to create one DAB per model and dataset. This approach aligns with best practices for modula...

You need to be sure you have an instance profile configured with PassRole permissions so that it can assume the cross account role to access the bucket and file notification resources. I found this technical blog helpful: https://community.databricks...

Hi @AntonPera , If the dropdown to change the profile type is disabled, you might need to create a new monitor from scratch. Here’s how you can do it: Go to the Lakehouse Monitoring section in Databricks.Create a new monitor and select the Snapshot p...

Hi @Pnascima , When using the Serverless cluster, what was the tshirt sizing? Looking at your issue with dedicated cluster, it sounds to me like a resource issue (hoping no data volume changes) You would have to find a comparable size of the interact...