- 1476 Views
- 2 replies
- 0 kudos
Error message while running queries
While running queries, both in SQL or notebooks, we get this error message below:INTERNAL_ERROR: Unexpected error when trying to access the statement result. Missing credentials to access the DBFS root storage container in Azure.The access connector ...
- 1476 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @CJOkpala This error suggests an issue with the credentials needed to access your Azure storage container from Databricks. Let's troubleshoot this methodically since there seems to be a disconnect between your configured access connector and the a...
- 0 kudos
- 1768 Views
- 2 replies
- 1 kudos
Copy a library into the folder of script ran in workflow job
I have a python script which gets run in a databricks workflow job task run using the git integration. Originally, in the repo contained a git submodule with a library (not supported by databricks). Therefore I need to copy the library repo (which I ...
- 1768 Views
- 2 replies
- 1 kudos
- 1 kudos
Thank you @lingareddy_Alva for your reply. Is it possible to have the path identification and copying done in a bash init script? I'd like to keep my run file clean as it is supposed to run also on other environments.
- 1 kudos
- 16516 Views
- 4 replies
- 1 kudos
Resolved! Databricks Job Timeout after 20 minutes
Hello,I have created a job with no timeout-seconds provided. But I am getting Error: Timed out within 20 minutes. I am running the below commands using Bash@3 task in ADO Pipeline yaml file. The code for the same is given belowtask: Bash@3 timeoutIn...
- 16516 Views
- 4 replies
- 1 kudos
- 1 kudos
I am also facing the same issue, can you please let me know where to change the default timeout ?
- 1 kudos
- 7732 Views
- 2 replies
- 0 kudos
Adding column as StructType
Hi Team,Just wondering, how can I add a column to an existing table.I'd tried the below script but giving me an error:ParseException: [PARSE_SYNTAX_ERROR] Syntax error at or near '<'(line 1, pos 121)ALTER TABLE table_clone ADD COLUMNS col_name1 STRUC...
- 7732 Views
- 2 replies
- 0 kudos
- 0 kudos
To add a STRUCT column to an existing table, use the correct syntax without $ symbols, such as ALTER TABLE table_clone ADD COLUMNS (col_name1 STRUCT<type: STRING, values: ARRAY<STRING>>)
- 0 kudos
- 2624 Views
- 4 replies
- 4 kudos
Is there a Databricks spark connector for java?
Is there a Databricks Spark connector for Java, just like we have for Snowflake (reference of Snowflake spark connector - https://docs.snowflake.com/en/user-guide/spark-connector-use)Essentially, the use case is to transfer data from S3 to a Databric...
- 2624 Views
- 4 replies
- 4 kudos
- 4 kudos
You don't need a separate Spark connector ,Databricks natively supports writing to Delta tables using standard Spark APIs. Instead of using JDBC, you can use df.write().format("delta") to efficiently write data from S3 to Databricks tables.
- 4 kudos
- 2556 Views
- 5 replies
- 1 kudos
Reading different file structures for json files in blob stores
Hi All,We are planning to store some mixed json files in blob store and read into Databricks. I am questioning whether we should have a container for each structure or if the various tools in Databricks can successfully read the different types. I ha...
- 2556 Views
- 5 replies
- 1 kudos
- 1 kudos
Organize files by schema into subfolders (e.g., /schema_type_a/, /schema_type_b/) in the same container.Avoid putting all JSON types in one folder
- 1 kudos
- 3811 Views
- 2 replies
- 0 kudos
Saving a file to /tmp is not working after migration to Unity Catalog
Hi,We upgraded our runtime cluster to Unity Catalog recently and since some of the code has been failing which was working fine earlier. We used to save files to "/tmp/" and then move them from temp into our blob storage however since the migration t...
- 3811 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @filipniziol Could we use volumes for this temp processing rather than doing S3
- 0 kudos
- 2066 Views
- 1 replies
- 0 kudos
Internal errors when running SQLs
We are running Databricks on GCP with a classic SQL warehouse. Its on the current version (v 2025.15)We have a pipeline that runs DBT on top of the SQL warehouseSince the 9th of May, our queries have been failing intermittently with internal errors f...
- 2066 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @utkarshamone The error messages you've shared—such as:-- [INTERNAL_ERROR] Query could not be scheduled: HTTP Response code: 503-- ExecutorLostFailure ... exited with code 134, sigabrt-- Internal error—indicate that your Databricks SQL warehouse o...
- 0 kudos
- 3061 Views
- 3 replies
- 1 kudos
Is there a Databricks spark connector for java?
Is there a Databricks Spark connector for Java, just like we have for Snowflake (reference of Snowflake spark connector - https://docs.snowflake.com/en/user-guide/spark-connector-use)Essentially, the use case is to transfer data from S3 to a Databric...
- 3061 Views
- 3 replies
- 1 kudos
- 1 kudos
Hey @I-am-Biplab , If running locally, it is going to be difficult to tune the performance up that much, but there are a few things you can try: 1. Up the partitions and batch size, as much as your machine will allow. Also, running repartition() coul...
- 1 kudos
- 13109 Views
- 11 replies
- 4 kudos
Resolved! ImportError: cannot import name 'AnalyzeArgument' from 'pyspark.sql.udtf'
Hello community,I installed databricks extension on my vscode ide. How to fix this error? I created the environment to run locally my notebooks and selected the available remote cluster to execute my notebook, what else?I Have this error: ImportError...
- 13109 Views
- 11 replies
- 4 kudos
- 4 kudos
@unj1m yes, as Alberto said you don't need to install pyspark, it is included in your cluster configuration.
- 4 kudos
- 1002 Views
- 1 replies
- 0 kudos
Resolved! Authentication issue in HiveMetastore
Problem Statement:When I execute the below code as a part of the notebook both manually and in workflow it works as expecteddf.write.mode("overwrite") \.format('delta') \.option('path',ext_path) \.saveAsTable("tbl_schema.Table_name")but when I integr...
- 1002 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Prajit0710 This is an interesting issue where your Delta table write operation works as expected when run directly,but when executed within a function, the table doesn't get recognized by the HiveMetastore.The key difference is likely related to ...
- 0 kudos
- 3808 Views
- 1 replies
- 0 kudos
can view allow_expectations_col in unit catalog
I am developing a dlt that manages expectations and it works correctly.but I need to see the columns__DROP_EXPECTATIONS_COL__MEETS_DROP_EXPECTATIONS__ALLOW_EXPECTATIONS_COLin the unified catalog, I can see them in the delta table that the dlt generat...
- 3808 Views
- 1 replies
- 0 kudos
- 0 kudos
Materialization tables created by DLT include these columns to process expectations but they might not propagate to Unity Catalog representations such as views or schema-level metadata unless explicitly set up for such lineage or column-level exposur...
- 0 kudos
- 4407 Views
- 1 replies
- 0 kudos
Unable to get s3 data - o536.ls.
Error while executingdisplay(dbutils.fs.ls(f"s3a://bucket-name/"))bucket-name has read/list permissionsshaded.databricks.org.apache.hadoop.fs.s3a.AWSClientIOException: getFileStatus on s3a://bucket-name/ com.amazonaws.SdkClientException: Unable to ex...
- 4407 Views
- 1 replies
- 0 kudos
- 0 kudos
To start with add SSL debugging logs by passing the JVM option -Djavax.net.debug=ssl in cluster configuration. This helps identify whether the handshake is failing due to missing certificates or invalid paths, Also check the cluster initialization sc...
- 0 kudos
- 4767 Views
- 1 replies
- 0 kudos
Error Listing Delta Log on GCS in Databricks
I am encountering an issue while working with a Delta table in Databricks. The error message is as follows:java.io.IOException: Error listing gs://cimb-prod-lakehouse/bronze-layer/dbd/customer_info_update_request_processing/_delta_log/ This issue occ...
- 4767 Views
- 1 replies
- 0 kudos
- 0 kudos
Ensure that the Databricks workspace has the necessary permissions to access the GCS bucket. Check if the service account used for Databricks has "Storage Object Viewer" or a similar role granted. Verify that the path "gs://cimb-prod-lakehouse/bronze...
- 0 kudos
- 4486 Views
- 2 replies
- 0 kudos
DLT Fails with Exception: CANNOT_READ_STREAMING_STATE_FILE
I have several DLT Pipeline, writing to some schema in a unity catalog. The storage location of the unity-catalog is managed by the databricks deployment (on AWS).The schema and the dlt-pipeline are managed via databricks asset bundles. I did not cha...
- 4486 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @DaPo , Have you made any code changes to your streaming query? There are limitations on what changes in a streaming query are allowed between restarts from the same checkpoint location. Refer this documentation The checkpoint location appears to ...
- 0 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 582 | |
| 349 | |
| 287 |
