- 5953 Views
- 5 replies
- 1 kudos
Executing an HTTP call from worker to localhost:80 (Unity Catalog)
hi,trying to implement this industry accelerator in UC (DBR 15.4, Shared Access Mode), I'm encountering an error when running an HTTP query (python requests) from within a udf (see this github issue). Are there any restrictions when it comes to run a...
- 5953 Views
- 5 replies
- 1 kudos
- 4027 Views
- 11 replies
- 1 kudos
Databricks SQL Warehouse Querys went to orphan state
We're experiencing an issue with our Databricks dbt workflow and workflow job is using the SQL warehouse L size cluster that's been working smoothly for the past couple of weeks. However, today we've noticed that at a specific time, all queries are g...
- 4027 Views
- 11 replies
- 1 kudos
- 8968 Views
- 2 replies
- 1 kudos
Cant connect power BI desktop to Azure databricks
Hello,I am trying to connect Power BI desktop to azure databricks (source: delta table) by downloading a connection file from Databricks. I see an error message like below when I open the connection file with power BI. Repeated attempts have given th...
- 8968 Views
- 2 replies
- 1 kudos
- 2729 Views
- 1 replies
- 0 kudos
VSCode Databricks-Connect can't find config file, says it doesn't exist, but it does
Hi all,I am getting an error that I previously didn't have within VSCode when authenticating to Databricks-Connect in PowerShell.When I run "databricks auth login" and choose a profile, I get the following:> Error: cannot load Databricks config file:...
- 2729 Views
- 1 replies
- 0 kudos
- 0 kudos
PrerequisitesDatabricks CLI installedMatch your databricks-connect version with your cluster runtime (e.g. runtime 14.3 LTS, needs databricks-connect 14.3.x)Match your local python installation with the Databricks Python version.Databricks extension ...
- 0 kudos
- 2838 Views
- 1 replies
- 0 kudos
ExecutorLostFailure
ExecutorLostFailure (executor 22 exited caused by one of the running tasks) Reason: Remote RPC client disassociated. Likely due to containers exceeding thresholds, or network issues. Check driver logs for WARN messages.
- 2838 Views
- 1 replies
- 0 kudos
- 0 kudos
he ExecutorLostFailure error in Spark indicates that an executor was lost during the execution of a task. Review the driver logs for any WARN or ERROR messages that might provide more context about why the executor was lost. Also, Ensure that the exe...
- 0 kudos
- 2715 Views
- 4 replies
- 2 kudos
Resolved! Differentiate null values in Variant Data type
Hello, Based on the documentation here, in both scenarios below try_variant_get returns a null: If the object cannot be foundif the object cannot be cast How does one differentiate between the two scenarios?
- 2715 Views
- 4 replies
- 2 kudos
- 2 kudos
Hi @vgautam, In the try_variant_get function, NULL is returned in two scenarios: Object Not Found: If the specified path does not exist in the JSON object.Invalid Cast: If the object at the specified path cannot be cast to the target type. To differe...
- 2 kudos
- 3682 Views
- 8 replies
- 2 kudos
Getting OOM error while processing xml data
I have a table in which one of the column contains xml raw data , approx. size of each row is 3MB, The volume of data is very huge, I have chunked it into 1 hour processing, On observing Memory Utilization metrics everything seems fine, but receiving...
- 3682 Views
- 8 replies
- 2 kudos
- 2 kudos
filteredDataframe=spark.table(f'{sourceConfig["srcDatabaseName"]}.{sourceConfig["srcTableName"]}').filter(f.col("load_dt")==current_start_time.date()).filter(f.col("load_ts")>=current_start_time).filter(f.col("load_ts")<current_end_time).filter("col...
- 2 kudos
- 5395 Views
- 11 replies
- 0 kudos
How to deploy unique workflows that running on production
Hello, community!I have a question about deploying workflows in a production environment. Specifically, how can we deploy a group of workflows to production so that they are created only once and cannot be duplicated by others?Currently, if someone d...
- 5395 Views
- 11 replies
- 0 kudos
- 0 kudos
I got some information from my internal team:The main thing to help here is deploying as a service principal and setting mode: production on the target. This is best done by setting up automation, such as Github Actions or Azure DevOps pipeline. You ...
- 0 kudos
- 2633 Views
- 2 replies
- 0 kudos
Unable to communicate with AWS DocumentDB
Runtime version: 15.4 LTS (includes Apache Spark 3.5.0, Scala 2.12)Spark config:'''spark.hadoop.datanucleus.fixedDatastore false spark.driver.extraJavaOptions -Djavax.net.ssl.trustStore=$JAVA_HOME/lib/security/cacerts spark.hadoop.javax.jdo.option.Co...
- 2633 Views
- 2 replies
- 0 kudos
- 0 kudos
The error seems to be related to writing data to a MongoDB data source, as indicated by the com.mongodb.spark.sql.connector.exceptions.DataException. It appears that the error is occurring during the execution of a Spark job that involves writing dat...
- 0 kudos
- 4863 Views
- 1 replies
- 0 kudos
Unable to read data from unity catalog, cannot store results of commands to DBFS
When attempting to read an excel (or any file) from a Unity Catalog volume (external or managed) the directory/file is not found. Additionally, when trying to use display() or any function like it to show results of a code chunk I am presented with"F...
- 4863 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @randythedataguy The error message "Failed to upload command result to DBFS. Error message: dbstoragekbbxxxxxxxxx.dfs.core.windows.net: Name or service not known" suggests a potential DNS resolution issue. Ensure that the DNS settings are correctl...
- 0 kudos
- 35227 Views
- 6 replies
- 0 kudos
How to delete Databricks Account
I created and used a Databricks Account for testing purposes. I want to delete that account. In the Databricks Account Web UI, there is no menu to delete an account. How should I delete it?
- 35227 Views
- 6 replies
- 0 kudos
- 0 kudos
Hi @17780 The easiest way is to delete the workspace and cancel your subscription.
- 0 kudos
- 995 Views
- 1 replies
- 0 kudos
Missing Rows When Reading Data from Impala Kudu to Databricks Using JDBC
Hi everyone,I’m working on a data ingestion process where I need to read data from an Impala Kudu table into Databricks using the JDBC connector. However, I’m experiencing an issue where some rows are missing in the data read. For instance, if there ...
- 995 Views
- 1 replies
- 0 kudos
- 0 kudos
@Kuke Have you checked whether the partitioning is configured correctly?If disabling partitioning (creating a single partition) allows you to retrieve 100,000 rows, but enabling partitioning results in only 99,000 rows, it is likely that the partitio...
- 0 kudos
- 4633 Views
- 5 replies
- 0 kudos
Resolved! How to Dynamically Retrieve Serverless Cluster ID for Databricks Job Configuration?
I am working on deploying a Databricks job to the production environment using a PowerShell script in Azure DevOps release pipeline. The task requires to update the job configuration JSON file to set the job's compute to serverless. For this, I need ...
- 4633 Views
- 5 replies
- 0 kudos
- 0 kudos
sure, happy to assist, if any of my responses was able to help you would really appreciate if you can accept it as a solution
- 0 kudos
- 4259 Views
- 2 replies
- 1 kudos
Delta tables and YOLO computer vision tasks
Hi all,I would really appreciate if someone could help me out. I feel it’s both a data engineering and ML question.One thing we use at wo is YOLO for object detection. I’ve managed to run YOLO by loading data from the blob storage, but I’ve seen tha...
- 4259 Views
- 2 replies
- 1 kudos
- 1 kudos
Hello @Andrewcon and @jnap , Have a look at Mosaic Streaming Dataset. You could load your data from your delta table and then train it on your PyTorch YOLO model. In that example, it use mobilenet model but you can adapt it to use YOLO. Petastorm is ...
- 1 kudos
- 3059 Views
- 0 replies
- 7 kudos
Best practices for writing efficient SQL queries in Databricks
Are your SQL queries running slow or consuming excessive resources? As data grows in size and complexity, writing optimized SQL queries becomes more crucial than ever. Knowing the right techniques can make all the difference.In my latest blog, I expl...
- 3059 Views
- 0 replies
- 7 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 581 | |
| 349 | |
| 287 |
