- 805 Views
- 1 replies
- 1 kudos
Resolved! Is it unusual that I need to start a compute cluster to sync with Git?
I would guess unusual but want to hear from others before I nag my managers about it. In Databricks (I access in web browser) we have a compute cluster specifically for Git; you need to start it to push code or even to change branches. This is separa...
- 805 Views
- 1 replies
- 1 kudos
- 1 kudos
It means you are on the old classic Git Proxy that helped establish connectivity from the Databricks Control Plane to your on-prem Git Server. If your Git Server was cloud-based you would not need the proxy cluster. That being said, the new way is th...
- 1 kudos
- 917 Views
- 1 replies
- 1 kudos
Resolved! Massive Duplicate Alerts Auto‑Created by Git Folders After Recent Databricks Update
Hello Databricks Support Team,I’m experiencing a severe issue in my Databricks workspace related to the new Git‑Integrated Alerts behavior. Overnight, my workspace went from 67 alerts to nearly 1,000 alerts, all of which appear to have been auto‑gene...
- 917 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @kcheng, Thanks for sharing the details. This looks like behaviour that will need workspace‑specific investigation by Databricks Support, rather than something the community can reliably diagnose or fix. Because it resulted in a sudden, large volu...
- 1 kudos
- 2672 Views
- 4 replies
- 1 kudos
Resolved! Checkpoint Location Error
Hi!I'm facing an error related to Checkpoint whenever I try to display a dataframe using auto Loader in Databricks free edition. Please refer the screenshot. To combat this, I have to delete the checkpoint folder and then execute the display or writ...
- 2672 Views
- 4 replies
- 1 kudos
- 1 kudos
Hi @AanchalSoni, I can’t see the full history of your notebook, so I’m not sure of the exact cause. But the behaviour strongly suggests that an earlier version of the stream used complete mode against the same checkpointLocation, and that configurati...
- 1 kudos
- 2397 Views
- 2 replies
- 2 kudos
Resolved! NULL rows getting inserted in delta table- Schema mismatch
I'm trying to add _metadata column while reading a json file: from pyspark.sql.functions import colfrom pyspark.sql.types import StructType, StructField, LongType, TimestampTypedf_accounts_read = spark.readStream.format("cloudFiles").\ option("clo...
- 2397 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @AanchalSoni, Looking at the first snapshot, it appears the path in all three records points to the checkpoint location. The _metadata column isn’t the root cause here. The issue is that Autoloader is ingesting your checkpoint files as data. Becau...
- 2 kudos
- 2248 Views
- 9 replies
- 3 kudos
Resolved! Partition cols for a temporary table in Lakefow SDP
Hi,I was going through the documentation on quarantining records. Initially I thought that partitioning is not supported for temporary tables however I came cross the following@DP.table( temporary=True, partition_cols=["is_quarantined"], ) @dp.ex...
- 2248 Views
- 9 replies
- 3 kudos
- 2250 Views
- 4 replies
- 0 kudos
Using spark.read.excel – dataAddress with only start cell is not working (DBR 17.x)
Hi, has anyone successfully used the Databricks Runtime 17.x native Excel reader with a dataAddress containing only a start cell (no end cell)? Even in the documentation, it is not specified (https://learn.microsoft.com/en-us/azure/databricks/query/f...
- 2250 Views
- 4 replies
- 0 kudos
- 0 kudos
Thanks @SteveOstrowski for the response. Yes I am using OPTION 1: USE A LARGE BOUNDING RANGE as a workaround.
- 0 kudos
- 1201 Views
- 2 replies
- 1 kudos
Resolved! Non-existent schema on redeployment of DAB with external volumes.
Hi all,DAB issue.My setup:Running CLI v0.294 on Python 3.12.11.Deployement is mode direct and using standard serverless compute.External locations in ADLS ST Container (container per ext loc).I'm attempting to deploy a bundle according to the followi...
- 1201 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @toast_2001, I did some digging and have a few helpful tips/tricks to assist your troubleshooting. So let me walk through what's likely happening and what to actually do about it. The error tells you that on the second deployment, DAB is trying to...
- 1 kudos
- 1037 Views
- 2 replies
- 0 kudos
Resolved! Lakebridge Reconcile Config not supporting "sfAuthenticator" for Snowflake
User is getting below error.Py4JJavaError: An error occurred while calling o508.load. : net.snowflake.client.jdbc.SnowflakeSQLException: Incorrect username or password was specified.https://github.com/databrickslabs/lakebridge/blob/201f9b7cae5ae45583...
- 1037 Views
- 2 replies
- 0 kudos
- 0 kudos
It seems the issue lies in the way Lakebridge is handling the Snowflake authentication, particularly with the sfAuthenticator parameter. While the plain Python code works, Lakebridge might not be correctly passing the necessary configuration for Snow...
- 0 kudos
- 1031 Views
- 2 replies
- 0 kudos
Resolved! Vacuum Command runs without any retention period even though the retention period was set
HelloI am trying to do tests on Vacuum in pytest.Command executed -> VACUUM unittest_mobi_edwhc_bul_replikation_001.t_bul_vacuum_experiment_1 RETAIN 0.05150017944444444 HOURSBut the Vacuum command is deleting all files and not considering the provide...
- 1031 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi Ashwin Many thanks for your answer. I also tested it in local pytest yesterday. The minimum supported value is 1 hour.
- 0 kudos
- 6141 Views
- 3 replies
- 1 kudos
Delta Live Table - Cannot redefine dataset
Hi,I am new to Delta Live Table.I am trying to create a delta live table from the databricks tutorial.I have created a notebook and attached an interactive cluster -DBR 14.3-LTS.I am running the below code.When I ran it for the 1st time it ran succes...
- 6141 Views
- 3 replies
- 1 kudos
- 1 kudos
@Walter_C there is no such "delete_table" API, see reference:https://learn.microsoft.com/en-us/azure/databricks/ldp/developer/python-ref#api-reference
- 1 kudos
- 3280 Views
- 12 replies
- 4 kudos
Databricks DBR 18.1 access workspace files error
import json with open("/Workspace/Users/<USER>/config.json", "r") as f: config = json.load(f) print(config)Throws following errorOSError: [Errno 5] Input/output error: '/Workspace/Users/<USER>/config.json'[Trace ID: 00-874e2bc3d747c3611c0c4a...
- 3280 Views
- 12 replies
- 4 kudos
- 1778 Views
- 1 replies
- 0 kudos
Resolved! Intermittent OSError: [Errno 5] Input/output error accessing workspace files from job
Every now and then, we get the following error in job cluster runs (when opening a workspace file):OSError: [Errno 5] Input/output error: '/Workspace/Users/<uuid>/.bundle/<path>'This happens even though the Python notebook (which is also in workspace...
- 1778 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Malthe, Sounds similar to this. Have you raised a support ticket? I can help raise it internally if there is a support ticket. If this answer resolves your question, could you mark it as “Accept as Solution”? That helps other users quickly find t...
- 0 kudos
- 30985 Views
- 21 replies
- 13 kudos
Databricks Bundles - How to select which jobs resources to deploy per target?
Hello, My team and I are experimenting with bundles, we follow the pattern of having one main file Databricks.yml and each job definition specified in a separate yaml for modularization. We wonder if it is possible to select from the main Databricks....
- 30985 Views
- 21 replies
- 13 kudos
- 13 kudos
One solution can be you can create separate databricks.yml file for each target such asqa/databricks.ymlprod/databricks.ymlqa,prod are folders named after target environmentHope this helps..
- 13 kudos
- 2275 Views
- 3 replies
- 2 kudos
Resolved! Workspace folder is visible but .py file cannot be read on job cluster (DBR 18)
Hi everyone,We are running into a strange issue when running notebooks on Databricks job clusters using DBR 18. It looks like the Workspace folder is mounted, but the .py file inside cannot be read immediately. I wanted to check if anyone else has ex...
- 2275 Views
- 3 replies
- 2 kudos
- 2 kudos
+1 to @pradeep_singh The Workspace FUSE (WSFS) daemons use ports 1015, 1017, and 1021 for communication between the driver and the executor. NFS tooling (hardcoded in glibc) can race with these ports during cluster startup, causing FUSE daemons to f...
- 2 kudos
- 3001 Views
- 6 replies
- 5 kudos
Resolved! Apache "Spark Connect"
Can someone confirm if this is the right message board for discussing the opensource Apache core of "Spark Connect". (aka databricks connect)We are hosting workloads on Azure Databricks, but would like to ensure that these workloads are following the...
- 3001 Views
- 6 replies
- 5 kudos
- 5 kudos
>> there is no native R UDF pathway over the wire. sparklyr works around this using rpy2, a Python library that embeds and executes R codeThis is interesting. I would not think of python as the best runtime for bridging. I'm wondering if this invol...
- 5 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 586 | |
| 349 | |
| 287 |
