- 672 Views
- 4 replies
- 2 kudos
Resolved! Differentiate null values in Variant Data type
Hello, Based on the documentation here, in both scenarios below try_variant_get returns a null: If the object cannot be foundif the object cannot be cast How does one differentiate between the two scenarios?
- 672 Views
- 4 replies
- 2 kudos
- 2 kudos
Hi @vgautam, In the try_variant_get function, NULL is returned in two scenarios: Object Not Found: If the specified path does not exist in the JSON object.Invalid Cast: If the object at the specified path cannot be cast to the target type. To differe...
- 2 kudos
- 1169 Views
- 8 replies
- 2 kudos
Getting OOM error while processing xml data
I have a table in which one of the column contains xml raw data , approx. size of each row is 3MB, The volume of data is very huge, I have chunked it into 1 hour processing, On observing Memory Utilization metrics everything seems fine, but receiving...
- 1169 Views
- 8 replies
- 2 kudos
- 2 kudos
filteredDataframe=spark.table(f'{sourceConfig["srcDatabaseName"]}.{sourceConfig["srcTableName"]}').filter(f.col("load_dt")==current_start_time.date()).filter(f.col("load_ts")>=current_start_time).filter(f.col("load_ts")<current_end_time).filter("col...
- 2 kudos
- 1016 Views
- 11 replies
- 0 kudos
How to deploy unique workflows that running on production
Hello, community!I have a question about deploying workflows in a production environment. Specifically, how can we deploy a group of workflows to production so that they are created only once and cannot be duplicated by others?Currently, if someone d...
- 1016 Views
- 11 replies
- 0 kudos
- 0 kudos
I got some information from my internal team:The main thing to help here is deploying as a service principal and setting mode: production on the target. This is best done by setting up automation, such as Github Actions or Azure DevOps pipeline. You ...
- 0 kudos
- 559 Views
- 2 replies
- 0 kudos
Unable to communicate with AWS DocumentDB
Runtime version: 15.4 LTS (includes Apache Spark 3.5.0, Scala 2.12)Spark config:'''spark.hadoop.datanucleus.fixedDatastore false spark.driver.extraJavaOptions -Djavax.net.ssl.trustStore=$JAVA_HOME/lib/security/cacerts spark.hadoop.javax.jdo.option.Co...
- 559 Views
- 2 replies
- 0 kudos
- 0 kudos
The error seems to be related to writing data to a MongoDB data source, as indicated by the com.mongodb.spark.sql.connector.exceptions.DataException. It appears that the error is occurring during the execution of a Spark job that involves writing dat...
- 0 kudos
- 1229 Views
- 1 replies
- 0 kudos
Unable to read data from unity catalog, cannot store results of commands to DBFS
When attempting to read an excel (or any file) from a Unity Catalog volume (external or managed) the directory/file is not found. Additionally, when trying to use display() or any function like it to show results of a code chunk I am presented with"F...
- 1229 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @randythedataguy The error message "Failed to upload command result to DBFS. Error message: dbstoragekbbxxxxxxxxx.dfs.core.windows.net: Name or service not known" suggests a potential DNS resolution issue. Ensure that the DNS settings are correctl...
- 0 kudos
- 29417 Views
- 6 replies
- 0 kudos
How to delete Databricks Account
I created and used a Databricks Account for testing purposes. I want to delete that account. In the Databricks Account Web UI, there is no menu to delete an account. How should I delete it?
- 29417 Views
- 6 replies
- 0 kudos
- 0 kudos
Hi @17780 The easiest way is to delete the workspace and cancel your subscription.
- 0 kudos
- 219 Views
- 1 replies
- 0 kudos
Missing Rows When Reading Data from Impala Kudu to Databricks Using JDBC
Hi everyone,I’m working on a data ingestion process where I need to read data from an Impala Kudu table into Databricks using the JDBC connector. However, I’m experiencing an issue where some rows are missing in the data read. For instance, if there ...
- 219 Views
- 1 replies
- 0 kudos
- 0 kudos
@Kuke Have you checked whether the partitioning is configured correctly?If disabling partitioning (creating a single partition) allows you to retrieve 100,000 rows, but enabling partitioning results in only 99,000 rows, it is likely that the partitio...
- 0 kudos
- 1197 Views
- 5 replies
- 0 kudos
Resolved! How to Dynamically Retrieve Serverless Cluster ID for Databricks Job Configuration?
I am working on deploying a Databricks job to the production environment using a PowerShell script in Azure DevOps release pipeline. The task requires to update the job configuration JSON file to set the job's compute to serverless. For this, I need ...
- 1197 Views
- 5 replies
- 0 kudos
- 0 kudos
sure, happy to assist, if any of my responses was able to help you would really appreciate if you can accept it as a solution
- 0 kudos
- 2516 Views
- 2 replies
- 1 kudos
Delta tables and YOLO computer vision tasks
Hi all,I would really appreciate if someone could help me out. I feel it’s both a data engineering and ML question.One thing we use at wo is YOLO for object detection. I’ve managed to run YOLO by loading data from the blob storage, but I’ve seen tha...
- 2516 Views
- 2 replies
- 1 kudos
- 1 kudos
Hello @Andrewcon and @jnap , Have a look at Mosaic Streaming Dataset. You could load your data from your delta table and then train it on your PyTorch YOLO model. In that example, it use mobilenet model but you can adapt it to use YOLO. Petastorm is ...
- 1 kudos
- 534 Views
- 0 replies
- 4 kudos
Best practices for writing efficient SQL queries in Databricks
Are your SQL queries running slow or consuming excessive resources? As data grows in size and complexity, writing optimized SQL queries becomes more crucial than ever. Knowing the right techniques can make all the difference.In my latest blog, I expl...
- 534 Views
- 0 replies
- 4 kudos
- 287 Views
- 0 replies
- 1 kudos
🚀 Simplifying Complex Data Pipelines with Databricks DLT
Are you dealing with complex XML inputs and wondering how to process them efficiently in a scalable data platform? Look no further! Delta Live Tables (DLT) in Databricks revolutionize how we handle complex data formats like XML. With its declarative...
- 287 Views
- 0 replies
- 1 kudos
- 20733 Views
- 6 replies
- 3 kudos
Resolved! Invalid Characters in Column Names " ,;{}()\n\t="
I'm reading data into a dataframe withdf = spark.read.json("s3://somepath/")I've tried first creating a delta table using the DeltaTable API with:DeltaTable.createIfNotExists(spark)\ .location(target_path)\ .addColumns(df.sche...
- 20733 Views
- 6 replies
- 3 kudos
- 54985 Views
- 4 replies
- 5 kudos
Best practices around bronze/silver/gold (medallion model) data lake classification?
What's the best way to organize our data lake and delta setup? We’re trying to use the bronze, silver and gold classification strategy. The main question is how do we know what classification the data is inside Databricks if there’s no actual physica...
- 54985 Views
- 4 replies
- 5 kudos
- 3045 Views
- 6 replies
- 1 kudos
Presto hive table to delta table conversion
Hi Everyone, I am using the below sql query to generate the days in order in hive & it is working fine. The table got migrated to delta and my query is failing. It would be appreciated if someone helps me to figure out the issue.SQL Query :with ex...
- 3045 Views
- 6 replies
- 1 kudos
- 1 kudos
Hi @SQL @jose_gonzalez , Have you tried code conversion tool fromTravinto technologies ? They have hive to delta table conversion
- 1 kudos
- 1292 Views
- 2 replies
- 0 kudos
Need guidance on connecting to Azure Databricks using JDBC Protocol
Step 1: Download and Reference the JDBC Driver Download the Databricks JDBC Driver: Visit the Databricks JDBC Driver download page. Download the appropriate version for your operating system. Extract the DatabricksJDBC42.jar file from the downloaded ...
- 1292 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @subhankar , Good Day! Looking at the error you are getting here shows that it tries to find some kind of JVM file and probably refers to the JAVA_HOME variable to achieve it. It looks as if it is not set correctly in your Environment Variables. ...
- 0 kudos
| User | Count |
|---|---|
| 1611 | |
| 768 | |
| 348 | |
| 286 | |
| 252 |
