- 1719 Views
- 1 replies
- 0 kudos
Massive increase in the number of "GetBlobProperties" operations
I had a massive increase in the volume of "GetBlobProperties" operations in my Azure Blob Storage account. The storage logs indicate that all the extra operations have IPs attributed to my Databricks resource group. I haven't made any changes to my r...
- 1719 Views
- 1 replies
- 0 kudos
- 0 kudos
Massive increase in "GetBlobProperties" operations in your Azure Blob Storage account could be due to the following1. Delta Tables and _delta_log Metadata AccessIf you're using Delta Lake, Databricks reads blob properties (e.g., last-modified time, s...
- 0 kudos
- 325 Views
- 0 replies
- 0 kudos
Embedding Dashboards on Databricks Apps
Hi Team,I recently tried the Hello World template and embedded the <iframe> from the dashboard that I created. It works properly fine before I added some code to have a Login Form (I used Dash Plotly on creating the Login Form) before the dashboard a...
- 325 Views
- 0 replies
- 0 kudos
- 547 Views
- 5 replies
- 4 kudos
create an external connection to oracle
Hi! I've been trying to create an external connection to oracle but getting the following error message "Detailed error message: ORA-00604: error occurred at recursive SQL level 1 ORA-01882: timezone region not found" I searched online and found some...
- 547 Views
- 5 replies
- 4 kudos
- 4 kudos
hey @austinoyoung ,I don't have an Oracle database to be able to test this for you, but I believe you can get around this error by following the steps laid out in here:https://stackoverflow.com/questions/9156379/ora-01882-timezone-region-not-foundIn ...
- 4 kudos
- 263 Views
- 1 replies
- 0 kudos
Autoloader Error Loading and Displaying
Hi there,I'd appreciate some assistance with troubleshooting what is supposed to be a (somewhat) simple use of autoloader. Below are some screenshots highlighting my issue:When I attempt to create the dataframe via spark.readStream.format("cloudFiles...
- 263 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @ChristianRRL This is a common issue with Spark Structured Streaming and the display() function.The error occurs because you're trying to display a streaming DataFrame, which requires special handling. Here are several solutions:1. Use writeStrea...
- 0 kudos
- 386 Views
- 0 replies
- 0 kudos
JDBC driver uses Unsafe API, which will be completely deprecated in a future release of Java
Using JDBC driver (2.7.3) in OpenJDK 24 gives the following warning:WARNING: A terminally deprecated method in sun.misc.Unsafe has been called WARNING: sun.misc.Unsafe::arrayBaseOffset has been called by com.databricks.client.jdbc42.internal.apache.a...
- 386 Views
- 0 replies
- 0 kudos
- 2076 Views
- 9 replies
- 5 kudos
Resolved! Materialized View Refresh - NUM_JOINS_THRESHOLD_EXCEEDED?
I have a very basic view with 3 inner joins that will only do a full refresh. Is there a limit to the number of joins you can have and still get an incremental refresh?"incrementalization_issues": [{"issue_type": "INCREMENTAL_PLAN_REJECTED_BY_COST_MO...
- 2076 Views
- 9 replies
- 5 kudos
- 5 kudos
@GregTyndall- how did you get those level of details (incrementalization_issues) for the MV build?
- 5 kudos
- 439 Views
- 5 replies
- 1 kudos
Resolved! Unable to install custom wheel in serverless environment
Hey guys, I have created a custom wheel to hold my common code. Since I cannot install task libraries on a serverless environment, I am installing this library in multiple notebooks using %pip install. What I do is I upload the library to a volume in...
- 439 Views
- 5 replies
- 1 kudos
- 1 kudos
@lezwon - Very interesting, as I have been wanting to do this and didn't attempt due to finding it was listed as not supported. Can you confirm what cloud provider you are using? AWS, Azure, GCP?
- 1 kudos
- 256 Views
- 3 replies
- 0 kudos
Resolved! vacuum does not work as expected
The delta.logRetentionDuration (default 30 Days) is generally not set on any table in my workspace. As per the documentation you can time travel within duration of log retention provided delta.deletedFileRetentionDuration also set for 30days. Which ...
- 256 Views
- 3 replies
- 0 kudos
- 0 kudos
this is preciously my observation after vacuuming. I do understand these 2 parameters, but its not working as expected. Even after vacuuming(retention for 30 days) we can go back 2 months and log are retained for more than 3 months
- 0 kudos
- 356 Views
- 4 replies
- 0 kudos
mapInPandas returning an intermittent error related to data type interconversion
```File "/databricks/spark/python/pyspark/sql/pandas/serializers.py", line 346, in _create_array return pa.Array.from_pandas( ^^^^^^^^^^^^^^^^^^^^^ File "pyarrow/array.pxi", line 1126, in pyarrow.lib.Array.from_pandas File "pyarrow/array.pxi", line 3...
- 356 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi @chinmay0924 Good day! Could you please confirm the following: Does the ID column incorrectly contain strings, which PyArrow fails to convert to integers (int64)?Are the data processed in both dataframes the exact same? Additionally, could you pro...
- 0 kudos
- 317 Views
- 2 replies
- 0 kudos
Resolved! SET a parameter in BEGIN END statement
Hello,How to set a parameter in a begin end statement. for exemple the following query fails : beginSET ansi_mode = true;end;with Cannot resolve variable `ANSI_MODE` on search path `SYSTEM`.`SESSION`. SQLSTATE: 42883
- 317 Views
- 2 replies
- 0 kudos
- 0 kudos
Hello @oneill There is currently no supported workaround to dynamically change system/session parameters such as ansi_mode within a BEGIN ... END block in Databricks SQL procedures or scripts. Can you set these parameters before executing any proced...
- 0 kudos
- 235 Views
- 2 replies
- 0 kudos
"with open" not working in single user access mode cluster (no such file or directory found)
Hi fellow engineers,So i was trying to read binary files (.jpg) in a ADLS2 mounted containerBut when im trying to read the file using "with open" i kept getting an error: No such file or directory foundI've read something related to this matter on So...
- 235 Views
- 2 replies
- 0 kudos
- 0 kudos
Weird...I'm able to access it without any issues. In case you are using community edition clusters, try copying the file to the driver node first and then read it.2nd option is below:
- 0 kudos
- 259 Views
- 2 replies
- 1 kudos
It's not going well to Connect to Amazon S3 with using Spark
I can't Connect to Amazon S3 well.I'm referencing and following this document: https://docs.databricks.com/gcp/en/connect/storage/amazon-s3But I can't access the S3 well.I believe the credentials are correct because I have verified that I can access ...
- 259 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi Isi,Thank you for your response — I really appreciate it Apologies, I didn’t explain my concern clearly.What I’m trying to confirm may be whether the instance profile overrides the spark.conf settings defined in a notebook.For example, I want to a...
- 1 kudos
- 1011 Views
- 1 replies
- 0 kudos
Announcing a new portfolio of Generative AI learning offerings on Databricks Academy Today, we launched new Generative AI, including LLMs, learning of...
Announcing a new portfolio of Generative AI learning offerings on Databricks AcademyToday, we launched new Generative AI, including LLMs, learning offerings for everyone from technical and business leaders to data practitioners, such as Data Scientis...
- 1011 Views
- 1 replies
- 0 kudos
- 0 kudos
Where can i find all the scripts / notebooks presented in the course for "Large Language Models (LLMs): Application through Production" ?
- 0 kudos
- 286 Views
- 1 replies
- 0 kudos
Inconsistent Decimal Comparison Behavior Between SQL Warehouse (Photon) and Spark Clusters
I'm seeing non-deterministic behavior when running the same query in SQL Warehouse (Photon) vs. interactive/job clusters (non-Photon), specifically involving a LEFT OUTER JOIN and a DECIMAL comparison in a WHERE clause. I have two views:View A: cont...
- 286 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @maarko This is a fascinating issue that points to several potential causes related to differences betweenPhoton and standard Spark execution engines, particularly around decimal handling and parallelism.Root Causes1. Decimal Precision and Scale ...
- 0 kudos
- 268 Views
- 1 replies
- 0 kudos
Lakeflow Connect - Column filtering
Hi community , I am interested in learning more about the feature that was mentioned in recent summit about query pushdown in lakeflow connect for SQL server. I believe this feature will allow to select only the required columns from source tables. I...
- 268 Views
- 1 replies
- 0 kudos
- 0 kudos
Hey @amitpm According to the documentation, this feature is currently in Public Preview, so if your Databricks account has access to public preview features, you can reach out to support to enable it and start testing performance.Setup guide for Lake...
- 0 kudos
| User | Count |
|---|---|
| 1614 | |
| 773 | |
| 349 | |
| 286 | |
| 253 |
