- 8021 Views
- 2 replies
- 1 kudos
Cant connect power BI desktop to Azure databricks
Hello,I am trying to connect Power BI desktop to azure databricks (source: delta table) by downloading a connection file from Databricks. I see an error message like below when I open the connection file with power BI. Repeated attempts have given th...
- 8021 Views
- 2 replies
- 1 kudos
- 1529 Views
- 1 replies
- 0 kudos
VSCode Databricks-Connect can't find config file, says it doesn't exist, but it does
Hi all,I am getting an error that I previously didn't have within VSCode when authenticating to Databricks-Connect in PowerShell.When I run "databricks auth login" and choose a profile, I get the following:> Error: cannot load Databricks config file:...
- 1529 Views
- 1 replies
- 0 kudos
- 0 kudos
PrerequisitesDatabricks CLI installedMatch your databricks-connect version with your cluster runtime (e.g. runtime 14.3 LTS, needs databricks-connect 14.3.x)Match your local python installation with the Databricks Python version.Databricks extension ...
- 0 kudos
- 1869 Views
- 1 replies
- 0 kudos
ExecutorLostFailure
ExecutorLostFailure (executor 22 exited caused by one of the running tasks) Reason: Remote RPC client disassociated. Likely due to containers exceeding thresholds, or network issues. Check driver logs for WARN messages.
- 1869 Views
- 1 replies
- 0 kudos
- 0 kudos
he ExecutorLostFailure error in Spark indicates that an executor was lost during the execution of a task. Review the driver logs for any WARN or ERROR messages that might provide more context about why the executor was lost. Also, Ensure that the exe...
- 0 kudos
- 1274 Views
- 4 replies
- 2 kudos
Resolved! Differentiate null values in Variant Data type
Hello, Based on the documentation here, in both scenarios below try_variant_get returns a null: If the object cannot be foundif the object cannot be cast How does one differentiate between the two scenarios?
- 1274 Views
- 4 replies
- 2 kudos
- 2 kudos
Hi @vgautam, In the try_variant_get function, NULL is returned in two scenarios: Object Not Found: If the specified path does not exist in the JSON object.Invalid Cast: If the object at the specified path cannot be cast to the target type. To differe...
- 2 kudos
- 2032 Views
- 8 replies
- 2 kudos
Getting OOM error while processing xml data
I have a table in which one of the column contains xml raw data , approx. size of each row is 3MB, The volume of data is very huge, I have chunked it into 1 hour processing, On observing Memory Utilization metrics everything seems fine, but receiving...
- 2032 Views
- 8 replies
- 2 kudos
- 2 kudos
filteredDataframe=spark.table(f'{sourceConfig["srcDatabaseName"]}.{sourceConfig["srcTableName"]}').filter(f.col("load_dt")==current_start_time.date()).filter(f.col("load_ts")>=current_start_time).filter(f.col("load_ts")<current_end_time).filter("col...
- 2 kudos
- 2211 Views
- 11 replies
- 0 kudos
How to deploy unique workflows that running on production
Hello, community!I have a question about deploying workflows in a production environment. Specifically, how can we deploy a group of workflows to production so that they are created only once and cannot be duplicated by others?Currently, if someone d...
- 2211 Views
- 11 replies
- 0 kudos
- 0 kudos
I got some information from my internal team:The main thing to help here is deploying as a service principal and setting mode: production on the target. This is best done by setting up automation, such as Github Actions or Azure DevOps pipeline. You ...
- 0 kudos
- 1563 Views
- 2 replies
- 0 kudos
Unable to communicate with AWS DocumentDB
Runtime version: 15.4 LTS (includes Apache Spark 3.5.0, Scala 2.12)Spark config:'''spark.hadoop.datanucleus.fixedDatastore false spark.driver.extraJavaOptions -Djavax.net.ssl.trustStore=$JAVA_HOME/lib/security/cacerts spark.hadoop.javax.jdo.option.Co...
- 1563 Views
- 2 replies
- 0 kudos
- 0 kudos
The error seems to be related to writing data to a MongoDB data source, as indicated by the com.mongodb.spark.sql.connector.exceptions.DataException. It appears that the error is occurring during the execution of a Spark job that involves writing dat...
- 0 kudos
- 3892 Views
- 1 replies
- 0 kudos
Unable to read data from unity catalog, cannot store results of commands to DBFS
When attempting to read an excel (or any file) from a Unity Catalog volume (external or managed) the directory/file is not found. Additionally, when trying to use display() or any function like it to show results of a code chunk I am presented with"F...
- 3892 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @randythedataguy The error message "Failed to upload command result to DBFS. Error message: dbstoragekbbxxxxxxxxx.dfs.core.windows.net: Name or service not known" suggests a potential DNS resolution issue. Ensure that the DNS settings are correctl...
- 0 kudos
- 32217 Views
- 6 replies
- 0 kudos
How to delete Databricks Account
I created and used a Databricks Account for testing purposes. I want to delete that account. In the Databricks Account Web UI, there is no menu to delete an account. How should I delete it?
- 32217 Views
- 6 replies
- 0 kudos
- 0 kudos
Hi @17780 The easiest way is to delete the workspace and cancel your subscription.
- 0 kudos
- 484 Views
- 1 replies
- 0 kudos
Missing Rows When Reading Data from Impala Kudu to Databricks Using JDBC
Hi everyone,I’m working on a data ingestion process where I need to read data from an Impala Kudu table into Databricks using the JDBC connector. However, I’m experiencing an issue where some rows are missing in the data read. For instance, if there ...
- 484 Views
- 1 replies
- 0 kudos
- 0 kudos
@Kuke Have you checked whether the partitioning is configured correctly?If disabling partitioning (creating a single partition) allows you to retrieve 100,000 rows, but enabling partitioning results in only 99,000 rows, it is likely that the partitio...
- 0 kudos
- 2691 Views
- 5 replies
- 0 kudos
Resolved! How to Dynamically Retrieve Serverless Cluster ID for Databricks Job Configuration?
I am working on deploying a Databricks job to the production environment using a PowerShell script in Azure DevOps release pipeline. The task requires to update the job configuration JSON file to set the job's compute to serverless. For this, I need ...
- 2691 Views
- 5 replies
- 0 kudos
- 0 kudos
sure, happy to assist, if any of my responses was able to help you would really appreciate if you can accept it as a solution
- 0 kudos
- 3223 Views
- 2 replies
- 1 kudos
Delta tables and YOLO computer vision tasks
Hi all,I would really appreciate if someone could help me out. I feel it’s both a data engineering and ML question.One thing we use at wo is YOLO for object detection. I’ve managed to run YOLO by loading data from the blob storage, but I’ve seen tha...
- 3223 Views
- 2 replies
- 1 kudos
- 1 kudos
Hello @Andrewcon and @jnap , Have a look at Mosaic Streaming Dataset. You could load your data from your delta table and then train it on your PyTorch YOLO model. In that example, it use mobilenet model but you can adapt it to use YOLO. Petastorm is ...
- 1 kudos
- 1531 Views
- 0 replies
- 4 kudos
Best practices for writing efficient SQL queries in Databricks
Are your SQL queries running slow or consuming excessive resources? As data grows in size and complexity, writing optimized SQL queries becomes more crucial than ever. Knowing the right techniques can make all the difference.In my latest blog, I expl...
- 1531 Views
- 0 replies
- 4 kudos
- 665 Views
- 0 replies
- 1 kudos
🚀 Simplifying Complex Data Pipelines with Databricks DLT
Are you dealing with complex XML inputs and wondering how to process them efficiently in a scalable data platform? Look no further! Delta Live Tables (DLT) in Databricks revolutionize how we handle complex data formats like XML. With its declarative...
- 665 Views
- 0 replies
- 1 kudos
- 27079 Views
- 6 replies
- 3 kudos
Resolved! Invalid Characters in Column Names " ,;{}()\n\t="
I'm reading data into a dataframe withdf = spark.read.json("s3://somepath/")I've tried first creating a delta table using the DeltaTable API with:DeltaTable.createIfNotExists(spark)\ .location(target_path)\ .addColumns(df.sche...
- 27079 Views
- 6 replies
- 3 kudos
| User | Count |
|---|---|
| 1619 | |
| 790 | |
| 485 | |
| 349 | |
| 287 |
