- 2246 Views
- 3 replies
- 1 kudos
Failed to checkout Git repository: RESOURCE_DOES_NOT_EXIST: Attempted to move non-existing node
I'm having issues with checking out Git repo in Workflows. Databricks can access files from commit `a` but fails to checkout the branch when attempting to access commit `b`. The error occurs specifically when trying to checkout commit `b`, and Databr...
- 2246 Views
- 3 replies
- 1 kudos
- 1 kudos
I didn't do anything to fix it. Databricks support did something to my workspace to fix the issue.
- 1 kudos
- 3625 Views
- 1 replies
- 0 kudos
Accidentally deleted files via API
Hello,I’m hoping you might be able to help me.I have accidentally deleted some Workspace files via API (an Azure DevOps code deployment pipeline). I can’t see the files in my Trash folder – are they gone forever, or is it possible to recover them on ...
- 3625 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @ohnomydata, Unfortunately files deleted via APIs or the Databricks CLI are permanently deleted and do not move to the Trash folder. The Trash folder is a UI-only feature, and items deleted through the UI can be recovered from the Trash within ...
- 0 kudos
- 1860 Views
- 2 replies
- 0 kudos
polars to natively read and write through adls
HI Everyone,Is there a way polars can directly read files from ADLS through abfss protocol.
- 1860 Views
- 2 replies
- 0 kudos
- 0 kudos
@pradeepvatsvk wrotae:HI Everyone,Is there a way polars can directly read files from ADLS through abfss protocol.Hello @pradeepvatsvk,Yes, Polars can directly read files from Azure Data Lake Storage (ADLS) using the ABFS (Azure Blob Filesystem) prot...
- 0 kudos
- 1517 Views
- 3 replies
- 0 kudos
Managed Delta Table corrupted
Hey guys,Recently, we have add some properties to our delta table and after that, the table shows error and we cannot do anything. The error is that: (java.util.NoSuchElementException) key not found: spark.sql.statistics.totalSizeI think maybe this i...
- 1517 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @Rafael-Sousa, Could you please raise a support case for this, to investigate this further? help@databricks.com
- 0 kudos
- 884 Views
- 1 replies
- 1 kudos
DAB multiple workspaces
Hi,We have 3 regional workspaces. Assume that we keep seperate folder for notebook say amer/xx , apac/xx, emea/xx and sepeate job/pipeline configrations for each region in git how to make sure during deploy appropriate job/pipleines are deployed in r...
- 884 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @samtech, Define separate bundle configuration files for each region. These configuration files will specify the resources (notebooks, jobs, pipelines) and their respective paths. For example, you can have amer_bundle.yml, apac_bundle.yml, and eme...
- 1 kudos
- 2308 Views
- 2 replies
- 0 kudos
create a one off job run using databricks SDK.
I'm trying to build the job spec using objects. When I try to call execute the job I get the following error.I'm somewhat new to python and not sure what I'm doing wrong here. Is anyone able to help?Traceback (most recent call last): File "y:\My ...
- 2308 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @BriGuy, Can you try importing this module first? from databricks.sdk.service.jobs import PermissionLevel
- 0 kudos
- 1452 Views
- 2 replies
- 0 kudos
Foreign Catalog refresh
Hi EveryoneI have recently created one Foreign catalog from AWS Redshift in databricks and I could see some tables too but when I ran REFRESH FOREIGN SCHEMA command, it failed with following error. I tried to search about it online but didn't get any...
- 1452 Views
- 2 replies
- 0 kudos
- 0 kudos
REFRESH FOREIGN SCHEMA is databricks command to refresh foreign catalog and I don't have visibility about the queries which its runs internally.
- 0 kudos
- 1247 Views
- 2 replies
- 0 kudos
When Try Returns Success for Invalid S3 Path in Spark: Is This a Bug?
Try(spark.read.format("parquet").load("s3://abcd/abcd/")) should result in Failure, but when executed in the notebook, it returns Success as shown below. Isn't this a bug?Try[DataFrame] = Success(...)
- 1247 Views
- 2 replies
- 0 kudos
- 0 kudos
@allinux The read is a valid way to load data. Why are you expecting a failure? can you please explain?
- 0 kudos
- 1344 Views
- 1 replies
- 1 kudos
Resolved! Several executions of a single notebook lead to java.lang.OutOfMemoryError
Hello,I am facing an issue that I do not understand. I have as simple Scala notebook with a "read function" that reads a json file on an external storage and does few changes to this DataFrame. I do my test on "all purpose" compute, DS3v2 (14gig/4cor...
- 1344 Views
- 1 replies
- 1 kudos
- 1 kudos
Finally, we understood the issue by ourself.By default, Databricks create new session for each new job. It is possible to change this behavior with the spark configuration (to put in spark config section of the compute settings):spark.databricks.sess...
- 1 kudos
- 12448 Views
- 5 replies
- 5 kudos
Spark issue handling data from json when the schema DataType mismatch occurs
Hi,I have encountered a problem using spark, when creating a dataframe from a raw json source.I have defined an schema for my data and the problem is that when there is a mismatch between one of the column values and its defined schema, spark not onl...
- 12448 Views
- 5 replies
- 5 kudos
- 5 kudos
@Farzad Bonabi​ :Thank you for reporting this issue. It seems to be a known bug in Spark when dealing with malformed decimal values. When a decimal value in the input JSON data is not parseable by Spark, it sets not only that column to null but also ...
- 5 kudos
- 1300 Views
- 4 replies
- 0 kudos
Error: from databricks import lakehouse_monitoring
We are using the following import: "from databricks import lakehouse_monitoring". We are receiving this error:ImportError: cannot import name 'lakehouse_monitoring' from 'databricks.sdk' (/databricks/python/lib/python3.11/site-packages/databricks/sdk...
- 1300 Views
- 4 replies
- 0 kudos
- 0 kudos
I wasnt able to find that module. However there are options under sdk. Refer to Lakehouse monitoring SDK reference %python import databricks print(dir(databricks.sdk))
- 0 kudos
- 1110 Views
- 1 replies
- 1 kudos
Lakeflow access
Hi,Can someone please advise how to sign up for Lakeflow access? I believe it's on public preview now, but it's not listed in my workspace's preview features list.Thanks!
- 1110 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @Fikrat, LakeFlow is currently in a gated Public Preview. To participate in the preview, you need to contact your Databricks account team. It is not listed in the workspace's preview features list because it requires specific access permissions th...
- 1 kudos
- 1529 Views
- 2 replies
- 0 kudos
How to Get the Size of Filtered Rows in Databricks SQL
I have a query that filters rows from a table based on a timestamp range. The query is as follows:SELECT COUNT(*) FROM table_name WHERE ts >= '2025-02-04 00:00:00' AND ts < '2025-02-05 00:00:00';This query returns 10 rows. I need to calculate the tot...
- 1529 Views
- 2 replies
- 0 kudos
- 0 kudos
@vidya_kothavale try this code block. Keep in mind to handle the null values.SELECT SUM(OCTET_LENGTH(CAST(column1 AS STRING)) + OCTET_LENGTH(CAST(column2 AS STRING)) + OCTET_LENGTH(CAST(COALESCE(column3, '0') AS STRING))) as bytes, SUM(OCTET...
- 0 kudos
- 1440 Views
- 2 replies
- 1 kudos
Urgent: Iceberg REST catalog - load a table has new JSON format
Hello, we are using Databricks Unity catalog to load an Iceberg table, i.e., Delta Lake table with Uniform feature). We are using this guide: https://docs.databricks.com/en/external-access/iceberg.html. This has been working for us since last year bu...
- 1440 Views
- 2 replies
- 1 kudos
- 1 kudos
Satyadeepak - it looks like this has been fixed on your end and we are no longer seeing this issue. Thanks for checking.
- 1 kudos
- 16513 Views
- 5 replies
- 1 kudos
DeltaFileNotFoundException: No file found in the directory (sudden task failure)
Hi all,I am currently running a job that will upsert a table by reading from delta change data feed from my silver table. Here is the relevent snippet of code: rds_changes = spark.read.format("delta") \ .option("readChangeFeed", "true") \ .optio...
- 16513 Views
- 5 replies
- 1 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 555 | |
| 349 | |
| 287 |
