Databricks Community

Hi @ElaPG1 , While the cluster sounds like a pretty good one with Autoscaling, it depends on the workload too. The Standard_D8s_v5 instances you are using have 32GB memory and 8 cores. While these are generally good, you might want to experiment with...

Hi @Garrus990 , To pass a parameter to a task that is based on a UNIX function, you can use the Databricks Jobs API to dynamically calculate dates with respect to the date of running your job.  Use a Notebook to Calculate Dates: Create a notebook tha...

Hi @mlopsuser , For organizing Databricks Asset Bundles (DABs) in your scenario with two separate regression models and datasets, it is generally recommended to create one DAB per model and dataset. This approach aligns with best practices for modula...

Make sure you are using the aggregates and SQL restrictions outlined in this article. https://docs.databricks.com/en/optimizations/incremental-refresh.htmlIf a SQL function is non-deterministic (current_timestamp() is a common one) you will have a CO...

You need to be sure you have an instance profile configured with PassRole permissions so that it can assume the cross account role to access the bucket and file notification resources. I found this technical blog helpful: https://community.databricks...

Hi @AntonPera , If the dropdown to change the profile type is disabled, you might need to create a new monitor from scratch. Here’s how you can do it: Go to the Lakehouse Monitoring section in Databricks.Create a new monitor and select the Snapshot p...

Hi @Pnascima , When using the Serverless cluster, what was the tshirt sizing? Looking at your issue with dedicated cluster, it sounds to me like a resource issue (hoping no data volume changes) You would have to find a comparable size of the interact...

Are you referring to the databricks-cli? If so, then are you probably sitting behind a firewall or proxy? If you are, then you may need to export the proxy settings in your terminal (export HTTP_PROXY=$proxy; export HTTPS_PROXY=$proxy, with their cor...

The SystemExit issue you’re seeing is typical with Click, as it’s designed for standalone CLI applications and automatically calls sys.exit() after running a command. This behavior can trigger SystemExit exceptions in non-CLI environments, like Datab...

The error you're encountering, [Databricks][JDBC](11500) Given type does not match given object: [Ljava.lang.String;@3e1346b0, indicates that the JDBC driver is not recognizing the Java String[] array as a valid SQL array type. This is a common issue...

The correlated subqueries within SQL User-Defined Functions (UDFs) used for row-level security are currently not supported for DELETE operations in Unity Catalog. You will need to adjust your row_filter_countryid_source_table UDF to avoid correlated ...

To resolve the ValueError: default auth: cannot configure default credentials error when using dbutils from WorkspaceClient in a Databricks notebook, follow these steps: Ensure SDK Installation: Make sure the Databricks SDK for Python is installed. ...

The error message is sometimes generic "possibly due to an OutOfMemoryError" There is memory pressure indeed, but try to correlate those graph metrics with the Driver's STDOUT file content and check if the GC/FullGCs are able to work properly and rec...